gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 4074:eb2f949ddfa2

Rename TimeGuesser to TimeParser.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 25 Jul 2019 15:08:11 +0200
parents	15c9d4064f0f
children	cb74aa69954e

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	19 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	22 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	23 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	25 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	26 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	28 "path/filepath"
4049 4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	29 "sort"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	30 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	31 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	32 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	34 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	35 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	39 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	40 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	51
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	52 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	53
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	57
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	58 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	62 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	63 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	68 agmStageDoneDeleteSQL = `
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	69 DELETE FROM waterway.gauge_measurements WHERE id IN (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	70 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	71 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	72 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	73 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	74 deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	75 )`
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	76
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	78 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	79 WHERE id IN (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	80 SELECT key
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	81 FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	82 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	83 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	84 NOT deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	85 )`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	86 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	87
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	89 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	93 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	94 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	95 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	96 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	98 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	99
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	100 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	101 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105
4074 eb2f949ddfa2 Rename TimeGuesser to TimeParser. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4060 diff changeset	106 var guessDate = misc.TimeParser([]string{
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	107 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	108 "2006-01-02T15:04:05-07:00",
4074 eb2f949ddfa2 Rename TimeGuesser to TimeParser. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4060 diff changeset	109 }).Parse
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	110
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	111 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	112
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	113 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	114 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	115 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	117 type agmLine struct {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	118 id int64
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	119 Location models.Isrs `json:"fk-gauge-id"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	120 CountryCode string `json:"country-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	121 Sender string `json:"sender"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	122 LanguageCode string `json:"language-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	123 DateIssue timetz `json:"date-issue"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	124 ReferenceCode string `json:"reference-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	125 MeasureDate timetz `json:"measure-date"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	126 WaterLevel float64 `json:"water-level"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	127 DateInfo timetz `json:"date-info"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	128 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	129 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	130
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	131 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	132 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	133 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	135 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	140
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	141 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	142 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	143 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	144 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	145 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	149 SELECT
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	150 id,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	155 reference_code,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	156 measure_date,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	161 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	162 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	163 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	164 AND measure_date BETWEEN $6 AND $7
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	165 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	166 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	167
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	168 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	169 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	171 validity,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	175 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	181 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	182 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	183 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	184 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	185 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	186 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	187 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	188 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	189 tstzrange(NULL, NULL)),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	200 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	201 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	206 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	207 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	208 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	209
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	210 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	211
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	212 headerFields := []struct {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	213 idx *int
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	214 name string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	215 }{
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	216 {fkGaugeIDIdx, "fk_gauge_id"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	217 {measureDateIdx, "measure_date"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	218 {valueIdx, "value"}, // "water_level",
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	219 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	220
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	221 nextHeader:
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	222 for i, f := range headers {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	223 h := strings.Replace(strings.ToLower(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	224 strings.TrimSpace(f)), " ", "_", -1)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	225
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	226 for j := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	227 if headerFields[j].name == h {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	228 if *headerFields[j].idx != -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	229 return fmt.Errorf(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	230 "There is more than one column namend '%s'", h)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	231 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	232 *headerFields[j].idx = i
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	233 continue nextHeader
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	234 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	235 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	236 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	237
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	238 var missing []string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	239 for i := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	240 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	241 missing = append(missing, headerFields[i].name)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	242 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	243 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	244 if len(missing) > 0 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	245 return fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	246 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	247
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	248 return nil
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	249 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	250
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	251 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	252 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	253 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	254 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	255 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	256 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	257 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	258
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	259 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	260
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	261 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	262 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	263 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	264 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	265 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	266
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	267 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	268 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	269 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	270
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	271 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	272 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	273 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	274 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	275
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	276 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	277 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	278 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	279 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	280 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	281
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	282 if err := parseAGMHeaders(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	283 headers,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	284 &fkGaugeIDIdx, &measureDateIdx, &valueIdx,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	285 ); err != nil {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	286 return nil, err
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	287 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	288
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	289 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	291 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	292 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	293 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	294
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	295 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	296 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	297 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	298 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	299 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	300
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	301 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	304 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	305 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	306
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	307 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	309 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	310 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	311 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	312
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	313 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	314
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	315 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	316
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	317 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100}
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	318 warn := warnLimiter.Warn
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	319 defer warnLimiter.Close()
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	320
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	321 agmLines := []*agmLine{}
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	322 ignored := 0
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	323 mdMinMax := map[models.Isrs]*[2]time.Time{}
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	324
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	325 lines:
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	326 for line := 1; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	328 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	329 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	330 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	331 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	332 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	333 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	334 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	335 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	336 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	337 }
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	338 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	339 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	340 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	341 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	342
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	343 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	344 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	345 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	346 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	347 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	348
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	349 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	350 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	351 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	352 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	353 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	354 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	355 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	356 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	357 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	358 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	359 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	360 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	361 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	362 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	363 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	364 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	365 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	366 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	367 if !exists {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	368 warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	369 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	370 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	371 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	372 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	373
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	374 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	375 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	376 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	377 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	378 if v := mdMinMax[*gid]; v != nil {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	379 if md.Before(v[0]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	380 v[0] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	381 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	382 if md.After(v[1]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	383 v[1] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	384 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	385 } else {
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	386 mdMinMax[*gid] = &[2]time.Time{md, md}
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	387 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	388
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	389 newSender := agm.Originator
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	390 newCountryCode := gid.CountryCode
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	391 newLanguageCode := misc.CCtoLang[gid.CountryCode]
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	392 newDateIssue := time.Now()
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	393 newReferenceCode := "ZPG"
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	394
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	395 value, err := strconv.ParseFloat(row[valueIdx], 32)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	396 if err != nil {
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	397 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	398 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	399 newValue := value
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	400
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	401 newDateInfo := newDateIssue
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	402
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	403 newSourceOrganization := newSender
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	404
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	405 agmLines = append(agmLines, newAGMLine(
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	406 *gid,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	407 newCountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	408 newSender,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	409 newLanguageCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	410 newDateIssue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	411 newReferenceCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	412 md,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	413 newValue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	414 newDateInfo,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	415 newSourceOrganization,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	416 ))
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	417 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	418
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	419 oldGMLines := map[models.Isrs]map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	420 for gid, minMax := range mdMinMax {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	421 oldGMLines[gid], err = getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	422 ctx, selectStmt, gid, minMax[0], minMax[1])
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	423 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	424 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	425 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	426 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	427
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	428 tx, err := conn.BeginTx(ctx, nil)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	429 if err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	430 return nil, err
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	431 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	432 defer tx.Rollback()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	433
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	434 txInsertStmt := tx.StmtContext(ctx, insertStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	435 txTrackStmt := tx.StmtContext(ctx, trackStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	436
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	437 agmLines:
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	438 for _, line := range agmLines {
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	439
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	440 var ase *agmSummaryEntry
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	441
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	442 if old := oldGMLines[line.Location]; old != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	443 ut := line.MeasureDate.Unix()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	444 if o, ok := old[ut]; ok {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	445 if !o.hasDiff(line) { // identical
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	446 // don't delete
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	447 delete(old, ut)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	448 continue agmLines
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	449 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	450 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	451 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	452 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	453 Versions: []*agmLine{o, line},
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	454 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	455 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	456 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	457 if ase == nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	458 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	459 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	460 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	461 Versions: []*agmLine{line},
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	462 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	463 }
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	464
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	465 var newID int64
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	466
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	467 if err := txInsertStmt.QueryRowContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	468 ctx,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	469 line.Location.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	470 line.Location.LoCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	471 line.Location.FairwaySection,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	472 line.Location.Orc,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	473 line.Location.Hectometre,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	474 line.MeasureDate.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	475 line.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	476 line.Sender,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	477 line.LanguageCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	478 line.DateIssue.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	479 line.ReferenceCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	480 line.WaterLevel,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	481 line.DateInfo.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	482 line.SourceOrganization,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	483 ).Scan(&newID); err != nil {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	484 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	485 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	486
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	487 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	488 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	489 newID,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	490 false,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	491 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	492 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	493 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	494
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	495 entries = append(entries, ase)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	496 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	497
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	498 var removed int
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	499
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	500 // Issue deletes
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	501 for _, old := range oldGMLines {
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	502 removed += len(old)
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	503 for _, line := range old {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	504 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	505 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	506 line.id,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	507 true,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	508 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	509 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	510 }
4049 4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	511 entries = append(entries, &agmSummaryEntry{
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	512 FKGaugeID: line.Location,
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	513 MeasureDate: line.MeasureDate,
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	514 Versions: []*agmLine{line, nil},
4a7c2140e44b AGM import: Generate entries for deleted items. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4037 diff changeset	515 })
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	516 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	517 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	518
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	519 feedback.Info("Measurements to update/insert: %d", len(entries))
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	520 feedback.Info("Measurements to delete: %d", removed)
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	521
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	522 if len(entries) == 0 && removed == 0 {
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	523 return nil, UnchangedError("No changes from AGM import")
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	524 }
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	525
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	526 if err = tx.Commit(); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	527 return nil, fmt.Errorf("Commit failed: %v", err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	528 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	529
4060 15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	530 // Sort here to mix the deletes right beside the matching inserts/updates.
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	531 // This also makes the output deterministic.
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	532 sort.Slice(entries, func(i, j int) bool {
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	533 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID)
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	534 })
15c9d4064f0f AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4049 diff changeset	535
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	536 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	537 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	538 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	539
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	540 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	541 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	542
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	543 func getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	544 ctx context.Context,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	545 stmt *sql.Stmt,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	546 location models.Isrs,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	547 from time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	548 to time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	549 ) (map[int64]*agmLine, error) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	550 var (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	551 oldID int64
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	552 oldCountryCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	553 oldSender string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	554 oldLanguageCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	555 oldDateIssue time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	556 oldReferenceCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	557 oldMeasureDate time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	558 oldValue float64
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	559 oldDateInfo time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	560 oldSourceOrganization string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	561 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	562 gmLines := map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	563
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	564 gms, err := stmt.QueryContext(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	565 ctx,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	566 location.CountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	567 location.LoCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	568 location.FairwaySection,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	569 location.Orc,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	570 location.Hectometre,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	571 from,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	572 to,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	573 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	574 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	575 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	576 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	577 defer gms.Close()
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	578 for gms.Next() {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	579 if err = gms.Scan(
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	580 &oldID,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	581 &oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	582 &oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	583 &oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	584 &oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	585 &oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	586 &oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	587 &oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	588 &oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	589 &oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	590 ); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	591 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	592 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	593 line := newAGMLine(
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	594 location,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	595 oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	596 oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	597 oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	598 oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	599 oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	600 oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	601 oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	602 oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	603 oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	604 )
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	605 line.id = oldID
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	606 gmLines[oldMeasureDate.Unix()] = line
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	607 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	608 if err = gms.Err(); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	609 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	610 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	611 return gmLines, nil
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	612 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	613
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	614 func newAGMLine(
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	615 location models.Isrs,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	616 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	617 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	618 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	619 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	620 referenceCode string,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	621 measureDate time.Time,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	622 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	623 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	624 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	625 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	626 return &agmLine{
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	627 Location: location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	628 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	629 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	630 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	631 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	632 ReferenceCode: referenceCode,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	633 MeasureDate: timetz{measureDate},
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	634 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	635 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	636 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	637 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	638 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 4074:eb2f949ddfa2