Mercurial > gemma

annotate pkg/imports/agm.go @ 2562:ce39e9954e85

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
author Sascha Wilde <wilde@intevation.de>
date Fri, 08 Mar 2019 18:57:58 +0100
parents de4dc3d16647
children 6d4f361c36e8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
13 // * Sascha Wilde <wilde@intevation.de>
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 import (
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
18 "bufio"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
20 "database/sql"
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
21 "encoding/csv"
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
22 "encoding/json"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
23 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
24 "io"
2173
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
25 "math"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 "os"
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
27 "path/filepath"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
28 "strconv"
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
29 "strings"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
30 "time"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
31
1938
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
32 "gemma.intevation.de/gemma/pkg/misc"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
33 "gemma.intevation.de/gemma/pkg/models"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
36 type ApprovedGaugeMeasurements struct {
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
37 Dir string `json:"dir"`
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
38 Originator string `json:"originator"`
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
41 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
42 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
49
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1743
diff changeset
50 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1743
diff changeset
51
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
54 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55
2187
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2173
diff changeset
56 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) }
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 func (agmJobCreator) Depends() []string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 return []string{
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 "gauges",
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61 "gauge_measurements",
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 const (
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
66 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
67 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
68 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
69 SELECT key
1995
59055c8301df Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents: 1938
diff changeset
70 FROM import.track_imports
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
71 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
72 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
73 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
74 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
75 SELECT o.id AS id
1781
0a53c33bc7b2 Fixed broken SQL statement (once again).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1780
diff changeset
76 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1780
diff changeset
77 JOIN waterway.gauge_measurements n
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
78 ON n.fk_gauge_id = o.fk_gauge_id AND n.measure_date = o.measure_date
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
79 WHERE n.id IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
80 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
81 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
82 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
83
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
84 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
85 UPDATE waterway.gauge_measurements SET staging_done = true
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
86 WHERE id IN (
1995
59055c8301df Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents: 1938
diff changeset
87 SELECT key FROM import.track_imports
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
88 WHERE import_id = $1 AND
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
89 relation = 'waterway.gauge_measurements'::regclass)`
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
90 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
91
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
92 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
93 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
96 ) error {
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
97 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
98 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
99 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
100 }
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
101 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
102 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
103
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
104 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
105 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
106 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
107 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
108 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
109
1938
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
110 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
111 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
112 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
113 }).Guess
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
114
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
115 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
116
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
117 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
118 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
119 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
120
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
121 type agmLine struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
122 CountryCode string `json:"country-code"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
123 Sender string `json:"sender"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
124 LanguageCode string `json:"language-code"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
125 DateIssue timetz `json:"date-issue"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
126 ReferenceCode string `json:"reference-code"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
127 WaterLevel float64 `json:"water-level"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
128 Predicted bool `json:"predicted"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
129 ValueMin *float64 `json:"value-min"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
130 ValueMax *float64 `json:"value-max"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
131 DateInfo timetz `json:"date-info"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
132 SourceOrganization string `json:"source-organization"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
133 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
134
2173
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
135 func (a *agmLine) hasDiff(b *agmLine) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
136 const eps = 0.00001
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
137 fdiff := func(x, y *float64) bool {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
138 if x == nil && y == nil {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
139 return false
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
140 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
141 if (x == nil && y != nil) || (x != nil && y == nil) {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
142 return true
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
143 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
144 return math.Abs(*x-*y) > eps
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
145 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
146 return a.CountryCode != b.CountryCode ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
147 a.Sender != b.Sender ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
148 a.LanguageCode != b.LanguageCode ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
149 a.ReferenceCode != b.ReferenceCode ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
150 math.Abs(a.WaterLevel-b.WaterLevel) > eps ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
151 a.Predicted != b.Predicted ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
152 fdiff(a.ValueMin, b.ValueMin) ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
153 fdiff(a.ValueMax, b.ValueMax) ||
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
154 a.SourceOrganization != b.SourceOrganization
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
155 }
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
156
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
157 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
158 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
159 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
160 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
161 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
162
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
163 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
164 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
165 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
166 id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
167 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
168 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
169 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
170 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
171 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
172 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
173 predicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
174 value_min,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
175 value_max,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
176 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
177 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
178 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
179 WHERE
2120
817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2105
diff changeset
180 fk_gauge_id = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) AND
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
181 measure_date = $6 AND staging_done`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
182
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
183 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
184 INSERT INTO waterway.gauge_measurements (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
185 fk_gauge_id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
186 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
187 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
188 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
189 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
190 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
191 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
192 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
193 predicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
194 value_min,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
195 value_max,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
196 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
197 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
198 is_waterlevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
199 staging_done
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
200 ) VALUES(
2120
817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2105
diff changeset
201 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
202 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
203 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
204 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
205 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
206 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
207 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
208 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
209 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
210 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
211 $15,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
212 $16,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
213 $17,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
214 true,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
215 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
216 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
217 RETURNING id`
2533
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
218
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
219 agmGaugeCheckSQL = `
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
220 SELECT EXISTS(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
221 SELECT 1 FROM waterway.gauges
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
222 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int))
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
223 `
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
224 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
225
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
226 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
227 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
228 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
229 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
230 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
231 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
232 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
233
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
234 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
235
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
236 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
237 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
238 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
239 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
240 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
241
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
242 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
243 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
244 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
245
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
246 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
247 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
248 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
249 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
250
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
251 var (
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
252 fkGaugeIDIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
253 measureDateIdx = -1
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
254 valueIdx = -1
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
255 )
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
256
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
257 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
258 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
259 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
260 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
261 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
262 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
263 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
264 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
265
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
266 nextHeader:
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
267 for i, f := range headers {
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
268 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
269 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
270
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
271 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
272 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
273 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
274 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
275 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
276 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
277 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
278 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
279 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
280 }
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
281 }
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
282
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
283 var missing []string
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
284 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
285 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
286 missing = append(missing, headerFields[i].name)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
287 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
288 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
289 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
290 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
291 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
292
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
293 tx, err := conn.BeginTx(ctx, nil)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
294 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
295 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
296 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
297 defer tx.Rollback()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
298
2533
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
299 gaugeCheckStmt, err := tx.PrepareContext(ctx, agmGaugeCheckSQL)
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
300 if err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
301 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
302 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
303 defer gaugeCheckStmt.Close()
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
304 selectStmt, err := tx.PrepareContext(ctx, agmSelectSQL)
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
305 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
306 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
307 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
308 defer selectStmt.Close()
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
309 insertStmt, err := tx.PrepareContext(ctx, agmInsertSQL)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
310 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
311 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
312 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
313 defer insertStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
314 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
315 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
316 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
317 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
318 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
319
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
320 entries := []*agmSummaryEntry{}
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
321
2533
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
322 checkedGauges := map[models.Isrs]bool{}
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
323
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
324 lines:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
325 for line := 1; ; line++ {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
326
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
327 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
328 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
329 case err == io.EOF || len(row) == 0:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
330 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
331 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
332 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
333 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
334
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
335 gids := row[fkGaugeIDIdx]
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
336 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
337 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
338 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
339 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
340
2533
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
341 if exists, found := checkedGauges[*gid]; found {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
342 if !exists {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
343 feedback.Warn("Ignoring data for unknown gauge %s", gid.String())
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
344 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
345 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
346 } else { // not found in gauge cache
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
347 if err := gaugeCheckStmt.QueryRowContext(
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
348 ctx,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
349 gid.CountryCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
350 gid.LoCode,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
351 gid.FairwaySection,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
352 gid.Orc,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
353 gid.Hectometre,
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
354 ).Scan(&exists); err != nil {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
355 return nil, err
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
356 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
357 checkedGauges[*gid] = exists
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
358 if !exists {
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
359 feedback.Warn("Ignoring data for unknown gauge %s", gid.String())
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
360 continue lines
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
361 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
362 }
de4dc3d16647 AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2187
diff changeset
363
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
364 md, err := guessDate(row[measureDateIdx])
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
365 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
366 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
367 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
368
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
369 var (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
370 oldID int64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
371 oldCountryCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
372 oldSender string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
373 oldLanguageCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
374 oldDateIssue time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
375 oldReferenceCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
376 oldValue float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
377 oldPredicted bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
378 oldValueMin sql.NullFloat64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
379 oldValueMax sql.NullFloat64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
380 oldDateInfo time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
381 oldSourceOrganization string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
382 )
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
383
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
384 err = selectStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
385 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
386 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
387 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
388 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
389 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
390 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
391 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
392 ).Scan(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
393 &oldID,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
394 &oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
395 &oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
396 &oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
397 &oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
398 &oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
399 &oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
400 &oldPredicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
401 &oldValueMin,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
402 &oldValueMax,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
403 &oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
404 &oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
405 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
406
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
407 var newEntry bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
408 switch {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
409 case err == sql.ErrNoRows:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
410 // Complete new one
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
411 newEntry = true
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
412 case err != nil:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
413 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
414 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
415
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
416 newSender := agm.Originator
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
417 newCountryCode := gid.CountryCode
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
418 newLanguageCode := misc.CCtoLang[gid.CountryCode]
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
419 newDateIssue := time.Now()
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
420 newReferenceCode := "ZPG"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
421
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
422 value, err := strconv.ParseFloat(row[valueIdx], 32)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
423 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
424 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
425 }
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
426 newValue := value
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
427
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
428 newPredicted := false
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
429
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
430 newValueMin := sql.NullFloat64{
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
431 Float64: 0,
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
432 Valid: true,
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
433 }
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
434 newValueMax := sql.NullFloat64{
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
435 Float64: 0,
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
436 Valid: true,
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
437 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
438
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
439 newDateInfo := newDateIssue
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
440
2562
ce39e9954e85 Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents: 2533
diff changeset
441 newSourceOrganization := newSender
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
442
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
443 var newID int64
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
444
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
445 if err := insertStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
446 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
447 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
448 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
449 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
450 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
451 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
452 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
453 newCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
454 newSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
455 newLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
456 newDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
457 newReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
458 newValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
459 newPredicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
460 newValueMin,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
461 newValueMax,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
462 newDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
463 newSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
464 ).Scan(&newID); err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
465 return nil, err
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
466 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
467 if _, err := trackStmt.ExecContext(
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
468 ctx, importID, "waterway.gauge_measurements", newID,
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
469 ); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
470 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
471 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
472
2173
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
473 n := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
474 newCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
475 newSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
476 newLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
477 newDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
478 newReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
479 newValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
480 newPredicted,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
481 newValueMin,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
482 newValueMax,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
483 newDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
484 newSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
485 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
486
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
487 ase := &agmSummaryEntry{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
488 FKGaugeID: *gid,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
489 MeasureDate: timetz{md},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
490 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
491
2173
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
492 if newEntry {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
493 ase.Versions = []*agmLine{n}
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
494 } else {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
495 o := newAGMLine(
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
496 oldCountryCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
497 oldSender,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
498 oldLanguageCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
499 oldDateIssue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
500 oldReferenceCode,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
501 oldValue,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
502 oldPredicted,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
503 oldValueMin,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
504 oldValueMax,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
505 oldDateInfo,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
506 oldSourceOrganization,
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
507 )
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
508 // Ignore if there is no diff.
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
509 if !n.hasDiff(o) {
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
510 continue
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
511 }
2173
1b1cb6cbfbf0 Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2168
diff changeset
512 ase.Versions = []*agmLine{o, n}
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
513 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
514 entries = append(entries, ase)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
515 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
516
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
517 if err := tx.Commit(); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
518 return nil, fmt.Errorf("Commit failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
519 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
520
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
521 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
522 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
523
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
524 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
525 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
526
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
527 func newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
528 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
529 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
530 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
531 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
532 referenceCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
533 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
534 predicted bool,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
535 valueMin sql.NullFloat64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
536 valueMax sql.NullFloat64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
537 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
538 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
539 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
540 nilFloat := func(v sql.NullFloat64) *float64 {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
541 var p *float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
542 if v.Valid {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
543 p = &v.Float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
544 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
545 return p
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
546 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
547 return &agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
548 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
549 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
550 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
551 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
552 ReferenceCode: referenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
553 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
554 Predicted: predicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
555 ValueMin: nilFloat(valueMin),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
556 ValueMax: nilFloat(valueMax),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
557 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
558 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
559 }
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
560 }