gemma: pkg/imports/agm.go annotate

annotate pkg/imports/agm.go @ 4037:a18bf6bc7e3c faster-agm

Detect unchanged case in AGM import.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Wed, 24 Jul 2019 11:59:36 +0200
parents	e45442db19b1
children	4a7c2140e44b

rev	line source
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	7 // Copyright (C) 2018, 2019 by via donau
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	13 // * Sascha Wilde <wilde@intevation.de>
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	14 // * Tom Gottfried <tom.gottfried@intevation.de>
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 import (
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	19 "bufio"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "database/sql"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	22 "encoding/csv"
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	23 "encoding/json"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	24 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	25 "io"
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	26 "math"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27 "os"
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	28 "path/filepath"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	29 "strconv"
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	30 "strings"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	31 "time"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	33 "gemma.intevation.de/gemma/pkg/misc"
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	34 "gemma.intevation.de/gemma/pkg/models"
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37 type ApprovedGaugeMeasurements struct {
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	38 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	39 Originator string `json:"originator"`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	42 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	43 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	44
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	45 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	46
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	48 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	51 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1743 diff changeset	52
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	53 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	54 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	55 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	56
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2173 diff changeset	57 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	59 func (agmJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	60 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	61 {"gauge_measurements"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3218 diff changeset	62 {"gauges"},
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66 const (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	67 agmStageDoneDeleteSQL = `
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	68 DELETE FROM waterway.gauge_measurements WHERE id IN (
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	69 SELECT key
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1938 diff changeset	70 FROM import.track_imports
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	71 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	72 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	73 deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	74 )`
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	75
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77 UPDATE waterway.gauge_measurements SET staging_done = true
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	78 WHERE id IN (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	79 SELECT key
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	80 FROM import.track_imports
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 WHERE import_id = $1 AND
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	82 relation = 'waterway.gauge_measurements'::regclass AND
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	83 NOT deletion
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	84 )`
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	86
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	87 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	89 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91 ) error {
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	92 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	93 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	94 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	95 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	98
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	99 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	100 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	101 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	102 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	103 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	104
1938 206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	105 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	106 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	107 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1781 diff changeset	108 }).Guess
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	109
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	110 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	111
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	112 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	113 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	114 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	115
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	116 type agmLine struct {
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	117 id int64
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	118 Location models.Isrs `json:"fk-gauge-id"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	119 CountryCode string `json:"country-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	120 Sender string `json:"sender"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	121 LanguageCode string `json:"language-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	122 DateIssue timetz `json:"date-issue"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	123 ReferenceCode string `json:"reference-code"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	124 MeasureDate timetz `json:"measure-date"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	125 WaterLevel float64 `json:"water-level"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	126 DateInfo timetz `json:"date-info"`
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	127 SourceOrganization string `json:"source-organization"`
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	128 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	129
2173 1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	130 func (a agmLine) hasDiff(b agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	131 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	132 return a.CountryCode != b.CountryCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	133 a.Sender != b.Sender \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	134 a.LanguageCode != b.LanguageCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	135 a.ReferenceCode != b.ReferenceCode \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	136 math.Abs(a.WaterLevel-b.WaterLevel) > eps \|\|
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	137 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	138 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	139
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	140 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	141 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	142 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	143 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	144 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	145
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	146 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	147 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	148 SELECT
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	149 id,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	150 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	151 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	152 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	153 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	154 reference_code,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	155 measure_date,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	156 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	157 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	158 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	159 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	160 WHERE
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	161 location
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	162 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	163 AND measure_date BETWEEN $6 AND $7
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	164 AND staging_done
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	165 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	166
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	167 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	168 INSERT INTO waterway.gauge_measurements (
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	169 location,
ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	170 validity,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	171 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	172 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	173 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	174 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	175 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	176 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	177 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	178 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	179 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	180 staging_done
3302 ec6163c6687d 'Historicise' gauges on import Tom Gottfried <tom@intevation.de> parents: 3278 diff changeset	181 ) VALUES (
2120 817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2105 diff changeset	182 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
3533 8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	183 COALESCE(
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	184 (SELECT validity FROM waterway.gauges
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	185 WHERE location
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	186 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	187 AND validity @> CAST($6 AS timestamp with time zone)),
8e083b271fca Improve error messages if no matching gauge version found Tom Gottfried <tom@intevation.de> parents: 3504 diff changeset	188 tstzrange(NULL, NULL)),
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	189 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	190 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	191 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	192 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	193 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	194 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	195 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	196 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	197 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	198 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	199 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	200 RETURNING id`
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	201
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	202 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	203 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	204 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	205 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	206 `
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	207 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	208
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	209 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	210
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	211 headerFields := []struct {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	212 idx *int
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	213 name string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	214 }{
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	215 {fkGaugeIDIdx, "fk_gauge_id"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	216 {measureDateIdx, "measure_date"},
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	217 {valueIdx, "value"}, // "water_level",
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	218 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	219
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	220 nextHeader:
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	221 for i, f := range headers {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	222 h := strings.Replace(strings.ToLower(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	223 strings.TrimSpace(f)), " ", "_", -1)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	224
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	225 for j := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	226 if headerFields[j].name == h {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	227 if *headerFields[j].idx != -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	228 return fmt.Errorf(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	229 "There is more than one column namend '%s'", h)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	230 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	231 *headerFields[j].idx = i
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	232 continue nextHeader
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	233 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	234 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	235 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	236
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	237 var missing []string
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	238 for i := range headerFields {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	239 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	240 missing = append(missing, headerFields[i].name)
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	241 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	242 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	243 if len(missing) > 0 {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	244 return fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	245 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	246
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	247 return nil
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	248 }
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	249
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	250 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	251 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	252 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	253 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	254 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	255 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	256 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	257
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	258 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	259
1743 85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	260 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	261 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	262 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	263 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	264 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	265
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	266 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	267 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	268 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	269
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	270 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	271 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	272 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	273 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1741 diff changeset	274
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	275 var (
2562 ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	276 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	277 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value" Sascha Wilde <wilde@intevation.de> parents: 2533 diff changeset	278 valueIdx = -1
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	279 )
1775 fcb0106ec510 Gauge measurement import: Added reference_code column. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1772 diff changeset	280
4030 040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	281 if err := parseAGMHeaders(
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	282 headers,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	283 &fkGaugeIDIdx, &measureDateIdx, &valueIdx,
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	284 ); err != nil {
040a5dc95eb9 AGM import: Moved parsing of CVS header line to separate function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4027 diff changeset	285 return nil, err
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	286 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	287
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	288 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL)
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	289 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	290 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	291 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	292 defer gaugeCheckStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	293
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	294 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL)
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	295 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	296 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	297 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	298 defer selectStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	299
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	300 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	301 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	302 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	303 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	304 defer insertStmt.Close()
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	305
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	306 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	307 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	308 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	309 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	310 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	311
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	312 entries := []*agmSummaryEntry{}
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	313
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	314 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	315
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	316 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100}
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	317 warn := warnLimiter.Warn
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	318 defer warnLimiter.Close()
6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	319
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	320 agmLines := []*agmLine{}
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	321 ignored := 0
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	322 mdMinMax := map[models.Isrs]*[2]time.Time{}
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	323
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	324 lines:
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	325 for line := 1; ; line++ {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	326
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	327 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	328 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	329 case err == io.EOF \|\| len(row) == 0:
3218 c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	330 feedback.Info("Read %d entries in CSV file", line-1)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	331 if ignored > 0 {
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	332 feedback.Info("%d entries ignored", ignored)
c2b65a549c6f Add minimal feedback in approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3193 diff changeset	333 }
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	334 if ignored == line-1 {
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	335 return nil, UnchangedError("No entries imported")
8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	336 }
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	337 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	338 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	339 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	340 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	341
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	342 gids := row[fkGaugeIDIdx]
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	343 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	344 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	345 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	346 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	347
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	348 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	349 if !exists {
3186 928cdc06ff37 Warn only once if gauge is unknown Tom Gottfried <tom@intevation.de> parents: 2853 diff changeset	350 // Just ignore the line since we have already warned
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	351 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	352 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	353 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	354 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	355 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	356 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	357 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	358 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	359 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	360 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	361 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	362 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	363 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	364 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	365 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	366 if !exists {
3504 6e748f31777a AGM/SR import. Limit logging of warnings to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3502 diff changeset	367 warn("Ignoring data for unknown gauge %s", gid.String())
3193 8329c6d3cf2a Do not offer an import for review that did not import anything Tom Gottfried <tom@intevation.de> parents: 3186 diff changeset	368 ignored++
2533 de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	369 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	370 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	371 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	372
2098 8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	373 md, err := guessDate(row[measureDateIdx])
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	374 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	375 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	376 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	377 if v := mdMinMax[*gid]; v != nil {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	378 if md.Before(v[0]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	379 v[0] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	380 }
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	381 if md.After(v[1]) {
f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	382 v[1] = md
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	383 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	384 } else {
4035 f2d5bf42ed38 AGM import: Simplified min/max measuredate handling a little bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4032 diff changeset	385 mdMinMax[*gid] = &[2]time.Time{md, md}
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	386 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	387
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	388 newSender := agm.Originator
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	389 newCountryCode := gid.CountryCode
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	390 newLanguageCode := misc.CCtoLang[gid.CountryCode]
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	391 newDateIssue := time.Now()
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	392 newReferenceCode := "ZPG"
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	393
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	394 value, err := strconv.ParseFloat(row[valueIdx], 32)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	395 if err != nil {
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	396 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	397 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	398 newValue := value
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	399
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	400 newDateInfo := newDateIssue
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	401
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	402 newSourceOrganization := newSender
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	403
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	404 agmLines = append(agmLines, newAGMLine(
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	405 *gid,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	406 newCountryCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	407 newSender,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	408 newLanguageCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	409 newDateIssue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	410 newReferenceCode,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	411 md,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	412 newValue,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	413 newDateInfo,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	414 newSourceOrganization,
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	415 ))
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	416 }
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	417
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	418 oldGMLines := map[models.Isrs]map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	419 for gid, minMax := range mdMinMax {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	420 oldGMLines[gid], err = getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	421 ctx, selectStmt, gid, minMax[0], minMax[1])
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	422 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	423 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	424 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	425 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	426
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	427 tx, err := conn.BeginTx(ctx, nil)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	428 if err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	429 return nil, err
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	430 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	431 defer tx.Rollback()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	432
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	433 txInsertStmt := tx.StmtContext(ctx, insertStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	434 txTrackStmt := tx.StmtContext(ctx, trackStmt)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	435
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	436 agmLines:
baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	437 for _, line := range agmLines {
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	438
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	439 var ase *agmSummaryEntry
1780 48791416bea5 (Approved) gauge measurement import: Fixed row level security. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1779 diff changeset	440
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	441 if old := oldGMLines[line.Location]; old != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	442 ut := line.MeasureDate.Unix()
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	443 if o, ok := old[ut]; ok {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	444 if !o.hasDiff(line) { // identical
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	445 // don't delete
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	446 delete(old, ut)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	447 continue agmLines
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	448 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	449 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	450 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	451 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	452 Versions: []*agmLine{o, line},
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	453 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	454 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	455 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	456 if ase == nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	457 ase = &agmSummaryEntry{
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	458 FKGaugeID: line.Location,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	459 MeasureDate: line.MeasureDate,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	460 Versions: []*agmLine{line},
3220 56b297592c0a Handle failing INSERTs gracefully during approved gauge measurements import Tom Gottfried <tom@intevation.de> parents: 3219 diff changeset	461 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	462 }
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	463
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	464 var newID int64
3502 45483dd0d801 AGM import: Don't pile up defer calls for every line in CSV file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3389 diff changeset	465
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	466 if err := txInsertStmt.QueryRowContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	467 ctx,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	468 line.Location.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	469 line.Location.LoCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	470 line.Location.FairwaySection,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	471 line.Location.Orc,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	472 line.Location.Hectometre,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	473 line.MeasureDate.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	474 line.CountryCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	475 line.Sender,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	476 line.LanguageCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	477 line.DateIssue.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	478 line.ReferenceCode,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	479 line.WaterLevel,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	480 line.DateInfo.Time,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	481 line.SourceOrganization,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	482 ).Scan(&newID); err != nil {
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	483 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	484 }
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	485
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	486 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	487 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	488 newID,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	489 false,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	490 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	491 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	492 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	493
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	494 entries = append(entries, ase)
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	495 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	496
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	497 var removed int
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	498
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	499 // Issue deletes
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	500 for _, old := range oldGMLines {
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	501 removed += len(old)
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	502 for _, line := range old {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	503 if _, err := txTrackStmt.ExecContext(
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	504 ctx, importID, "waterway.gauge_measurements",
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	505 line.id,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	506 true,
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	507 ); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	508 return nil, err
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	509 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	510 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	511 }
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	512
4037 a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	513 feedback.Info("Measurements to update/insert: %d", len(entries))
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	514 feedback.Info("Measurements to delete: %d", removed)
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	515
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	516 if len(entries) == 0 && removed == 0 {
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	517 return nil, UnchangedError("No changes from AGM import")
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	518 }
a18bf6bc7e3c Detect unchanged case in AGM import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4036 diff changeset	519
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	520 if err = tx.Commit(); err != nil {
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	521 return nil, fmt.Errorf("Commit failed: %v", err)
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	522 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	523
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	524 feedback.Info("Imported %d entries with changes", len(entries))
1778 164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	525 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	526 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1775 diff changeset	527
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	528 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	529 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	530
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	531 func getOldGMLines(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	532 ctx context.Context,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	533 stmt *sql.Stmt,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	534 location models.Isrs,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	535 from time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	536 to time.Time,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	537 ) (map[int64]*agmLine, error) {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	538 var (
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	539 oldID int64
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	540 oldCountryCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	541 oldSender string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	542 oldLanguageCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	543 oldDateIssue time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	544 oldReferenceCode string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	545 oldMeasureDate time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	546 oldValue float64
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	547 oldDateInfo time.Time
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	548 oldSourceOrganization string
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	549 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	550 gmLines := map[int64]*agmLine{}
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	551
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	552 gms, err := stmt.QueryContext(
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	553 ctx,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	554 location.CountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	555 location.LoCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	556 location.FairwaySection,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	557 location.Orc,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	558 location.Hectometre,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	559 from,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	560 to,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	561 )
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	562 if err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	563 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	564 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	565 defer gms.Close()
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	566 for gms.Next() {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	567 if err = gms.Scan(
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	568 &oldID,
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	569 &oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	570 &oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	571 &oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	572 &oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	573 &oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	574 &oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	575 &oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	576 &oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	577 &oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	578 ); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	579 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	580 }
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	581 line := newAGMLine(
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	582 location,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	583 oldCountryCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	584 oldSender,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	585 oldLanguageCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	586 oldDateIssue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	587 oldReferenceCode,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	588 oldMeasureDate,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	589 oldValue,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	590 oldDateInfo,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	591 oldSourceOrganization,
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	592 )
4036 e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	593 line.id = oldID
e45442db19b1 First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4035 diff changeset	594 gmLines[oldMeasureDate.Unix()] = line
4027 b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	595 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	596 if err = gms.Err(); err != nil {
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	597 return nil, err
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	598 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	599 return gmLines, nil
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	600 }
b17453420eff Avoid doing one SELECT per line in import file Tom Gottfried <tom@intevation.de> parents: 4026 diff changeset	601
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	602 func newAGMLine(
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	603 location models.Isrs,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	604 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	605 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	606 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	607 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	608 referenceCode string,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	609 measureDate time.Time,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	610 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	611 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	612 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	613 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	614 return &agmLine{
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	615 Location: location,
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	616 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	617 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	618 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	619 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	620 ReferenceCode: referenceCode,
4023 baa51bb82364 AGM import: parse CSV to completion before requesting database Tom Gottfried <tom@intevation.de> parents: 3533 diff changeset	621 MeasureDate: timetz{measureDate},
2105 58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	622 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	623 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	624 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2098 diff changeset	625 }
1741 44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	626 }

Mercurial > gemma

annotate pkg/imports/agm.go @ 4037:a18bf6bc7e3c faster-agm