Mercurial > gemma
annotate pkg/imports/agm.go @ 2533:de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Thu, 07 Mar 2019 11:57:05 +0100 |
parents | 7c83b5277c1c |
children | ce39e9954e85 |
rev | line source |
---|---|
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
17 "bufio" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "context" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "database/sql" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
20 "encoding/csv" |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
21 "encoding/json" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
22 "fmt" |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
23 "io" |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
24 "math" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "os" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
26 "path/filepath" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
27 "strconv" |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
28 "strings" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
29 "time" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
31 "gemma.intevation.de/gemma/pkg/misc" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
32 "gemma.intevation.de/gemma/pkg/models" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
34 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
35 type ApprovedGaugeMeasurements struct { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 Dir string `json:"dir"` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
39 // GMAPJobKind is the unique name of an approved gauge measurements import job. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
40 const AGMJobKind JobKind = "agm" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 type agmJobCreator struct{} |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 func init() { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 RegisterJobCreator(AGMJobKind, agmJobCreator{}) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
48 func (agmJobCreator) AutoAccept() bool { return false } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
49 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 func (agmJobCreator) Description() string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 return "approved gauge measurements" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2173
diff
changeset
|
54 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 func (agmJobCreator) Depends() []string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 return []string{ |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 "gauges", |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 "gauge_measurements", |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 const ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
64 // delete the old and keep the new measures. |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
65 agmStageDoneDeleteSQL = ` |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
66 WITH staged AS ( |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
67 SELECT key |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
68 FROM import.track_imports |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
69 WHERE import_id = $1 AND |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
70 relation = 'waterway.gauge_measurements'::regclass |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
71 ), |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
72 to_delete AS ( |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
73 SELECT o.id AS id |
1781
0a53c33bc7b2
Fixed broken SQL statement (once again).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1780
diff
changeset
|
74 FROM waterway.gauge_measurements o |
0a53c33bc7b2
Fixed broken SQL statement (once again).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1780
diff
changeset
|
75 JOIN waterway.gauge_measurements n |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
76 ON n.fk_gauge_id = o.fk_gauge_id AND n.measure_date = o.measure_date |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
77 WHERE n.id IN (SELECT key FROM staged) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
78 AND o.id NOT IN (SELECT key FROM staged) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
79 ) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
80 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)` |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
81 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 agmStageDoneSQL = ` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 UPDATE waterway.gauge_measurements SET staging_done = true |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
84 WHERE id IN ( |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
85 SELECT key FROM import.track_imports |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 WHERE import_id = $1 AND |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
87 relation = 'waterway.gauge_measurements'::regclass)` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 func (agmJobCreator) StageDone( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 tx *sql.Tx, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 id int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 ) error { |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
95 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
96 if err == nil { |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
97 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
98 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 return err |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 // CleanUp removes the folder containing the CSV file with the |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 // the approved gauge measurements. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 func (agm *ApprovedGaugeMeasurements) CleanUp() error { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 return os.RemoveAll(agm.Dir) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
108 var guessDate = misc.TimeGuesser([]string{ |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
109 "02.01.2006 15:04", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
110 "2006-01-02T15:04:05-07:00", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
111 }).Guess |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
112 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
113 type timetz struct{ time.Time } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
114 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
115 func (ttz *timetz) MarshalJSON() ([]byte, error) { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
116 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00")) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
117 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
118 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
119 type agmLine struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
120 CountryCode string `json:"country-code"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
121 Sender string `json:"sender"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
122 LanguageCode string `json:"language-code"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
123 DateIssue timetz `json:"date-issue"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
124 ReferenceCode string `json:"reference-code"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
125 WaterLevel float64 `json:"water-level"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
126 Predicted bool `json:"predicted"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
127 ValueMin *float64 `json:"value-min"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
128 ValueMax *float64 `json:"value-max"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
129 DateInfo timetz `json:"date-info"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
130 SourceOrganization string `json:"source-organization"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
131 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
132 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
133 func (a *agmLine) hasDiff(b *agmLine) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
134 const eps = 0.00001 |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
135 fdiff := func(x, y *float64) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
136 if x == nil && y == nil { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
137 return false |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
138 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
139 if (x == nil && y != nil) || (x != nil && y == nil) { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
140 return true |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
141 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
142 return math.Abs(*x-*y) > eps |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
143 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
144 return a.CountryCode != b.CountryCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
145 a.Sender != b.Sender || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
146 a.LanguageCode != b.LanguageCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
147 !a.DateIssue.Time.Equal(b.DateIssue.Time) || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
148 a.ReferenceCode != b.ReferenceCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
149 math.Abs(a.WaterLevel-b.WaterLevel) > eps || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
150 a.Predicted != b.Predicted || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
151 fdiff(a.ValueMin, b.ValueMin) || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
152 fdiff(a.ValueMax, b.ValueMax) || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
153 !a.DateInfo.Time.Equal(b.DateInfo.Time) || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
154 a.SourceOrganization != b.SourceOrganization |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
155 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
156 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
157 type agmSummaryEntry struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
158 FKGaugeID models.Isrs `json:"fk-gauge-id"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
159 MeasureDate timetz `json:"measure-date"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
160 Versions []*agmLine `json:"versions"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
161 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
162 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
163 const ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
164 agmSelectSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
165 SELECT |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
166 id, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
167 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
168 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
169 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
170 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
171 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
172 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
173 predicted, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
174 value_min, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
175 value_max, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
176 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
177 source_organization |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
178 FROM waterway.gauge_measurements |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
179 WHERE |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
180 fk_gauge_id = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) AND |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
181 measure_date = $6 AND staging_done` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
182 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
183 agmInsertSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
184 INSERT INTO waterway.gauge_measurements ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
185 fk_gauge_id, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
186 measure_date, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
187 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
188 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
189 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
190 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
191 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
192 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
193 predicted, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
194 value_min, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
195 value_max, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
196 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
197 source_organization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
198 is_waterlevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
199 staging_done |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
200 ) VALUES( |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
201 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
202 $6, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
203 $7, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
204 $8, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
205 $9, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
206 $10, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
207 $11, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
208 $12, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
209 $13, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
210 $14, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
211 $15, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
212 $16, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
213 $17, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
214 true, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
215 false |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
216 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
217 RETURNING id` |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
218 |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
219 agmGaugeCheckSQL = ` |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
220 SELECT EXISTS( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
221 SELECT 1 FROM waterway.gauges |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
222 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
223 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
224 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
225 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
226 // Do executes the actual approved gauge measurements import. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
227 func (agm *ApprovedGaugeMeasurements) Do( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
228 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
229 importID int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
230 conn *sql.Conn, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
231 feedback Feedback, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
232 ) (interface{}, error) { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
233 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
234 start := time.Now() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
235 |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
236 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv")) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
237 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
238 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
239 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
240 defer f.Close() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
241 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
242 r := csv.NewReader(bufio.NewReader(f)) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
243 r.Comma = ';' |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
244 r.ReuseRecord = true |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
245 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
246 headers, err := r.Read() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
247 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
248 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
249 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
250 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
251 var ( |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
252 fkGaugeIDIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
253 measureDateIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
254 fromIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
255 languageCodeIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
256 countryCodeIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
257 dateIssueIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
258 referenceCodeIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
259 valueIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
260 predictedIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
261 valueMinIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
262 valueMaxIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
263 dateInfoIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
264 originatorIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
265 unitIdx = -1 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
266 ) |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
267 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
268 headerFields := []struct { |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
269 idx *int |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
270 name string |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
271 }{ |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
272 {&fkGaugeIDIdx, "fk_gauge_id"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
273 {&measureDateIdx, "measure_date"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
274 {&fromIdx, "from"}, // "sender", |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
275 {&languageCodeIdx, "language_code"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
276 {&countryCodeIdx, "country_code"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
277 {&dateIssueIdx, "date_issue"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
278 {&referenceCodeIdx, "reference_code"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
279 {&valueIdx, "value"}, // "water_level", |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
280 {&predictedIdx, "predicted"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
281 // "is_waterlevel", |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
282 {&valueMinIdx, "value_min"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
283 {&valueMaxIdx, "value_max"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
284 {&dateInfoIdx, "date_info"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
285 {&originatorIdx, "originator"}, // "source_organization", |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
286 {&unitIdx, "unit"}, |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
287 } |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
288 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
289 nextHeader: |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
290 for i, f := range headers { |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
291 h := strings.Replace(strings.ToLower( |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
292 strings.TrimSpace(f)), " ", "_", -1) |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
293 |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
294 for j := range headerFields { |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
295 if headerFields[j].name == h { |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
296 if *headerFields[j].idx != -1 { |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
297 return nil, fmt.Errorf( |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
298 "There is more than one column namend '%s'", h) |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
299 } |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
300 *headerFields[j].idx = i |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
301 continue nextHeader |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
302 } |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
303 } |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
304 } |
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
305 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
306 var missing []string |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
307 for i := range headerFields { |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
308 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 { |
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
309 missing = append(missing, headerFields[i].name) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
310 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
311 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
312 if len(missing) > 0 { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
313 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", ")) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
314 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
315 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
316 inCm, _ := rescale("cm") |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
317 scaler := func(row []string) (func(float32) float32, error) { |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
318 if unitIdx == -1 { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
319 return inCm, nil |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
320 } |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
321 unit := row[unitIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
322 if unit == "cm" { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
323 return inCm, nil |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
324 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
325 s, err := rescale(unit) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
326 return s, err |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
327 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
328 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
329 tx, err := conn.BeginTx(ctx, nil) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
330 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
331 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
332 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
333 defer tx.Rollback() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
334 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
335 gaugeCheckStmt, err := tx.PrepareContext(ctx, agmGaugeCheckSQL) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
336 if err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
337 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
338 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
339 defer gaugeCheckStmt.Close() |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
340 selectStmt, err := tx.PrepareContext(ctx, agmSelectSQL) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
341 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
342 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
343 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
344 defer selectStmt.Close() |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
345 insertStmt, err := tx.PrepareContext(ctx, agmInsertSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
346 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
347 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
348 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
349 defer insertStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
350 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
351 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
352 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
353 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
354 defer trackStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
355 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
356 entries := []*agmSummaryEntry{} |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
357 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
358 checkedGauges := map[models.Isrs]bool{} |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
359 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
360 lines: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
361 for line := 1; ; line++ { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
362 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
363 row, err := r.Read() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
364 switch { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
365 case err == io.EOF || len(row) == 0: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
366 break lines |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
367 case err != nil: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
368 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
369 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
370 convert, err := scaler(row) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
371 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
372 return nil, fmt.Errorf("line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
373 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
374 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
375 gids := row[fkGaugeIDIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
376 gid, err := models.IsrsFromString(gids) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
377 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
378 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
379 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
380 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
381 if exists, found := checkedGauges[*gid]; found { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
382 if !exists { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
383 feedback.Warn("Ignoring data for unknown gauge %s", gid.String()) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
384 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
385 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
386 } else { // not found in gauge cache |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
387 if err := gaugeCheckStmt.QueryRowContext( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
388 ctx, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
389 gid.CountryCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
390 gid.LoCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
391 gid.FairwaySection, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
392 gid.Orc, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
393 gid.Hectometre, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
394 ).Scan(&exists); err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
395 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
396 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
397 checkedGauges[*gid] = exists |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
398 if !exists { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
399 feedback.Warn("Ignoring data for unknown gauge %s", gid.String()) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
400 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
401 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
402 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
403 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
404 md, err := guessDate(row[measureDateIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
405 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
406 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
407 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
408 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
409 var ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
410 oldID int64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
411 oldCountryCode string |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
412 oldSender string |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
413 oldLanguageCode string |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
414 oldDateIssue time.Time |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
415 oldReferenceCode string |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
416 oldValue float64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
417 oldPredicted bool |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
418 oldValueMin sql.NullFloat64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
419 oldValueMax sql.NullFloat64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
420 oldDateInfo time.Time |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
421 oldSourceOrganization string |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
422 ) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
423 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
424 err = selectStmt.QueryRowContext( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
425 ctx, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
426 gid.CountryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
427 gid.LoCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
428 gid.FairwaySection, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
429 gid.Orc, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
430 gid.Hectometre, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
431 md, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
432 ).Scan( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
433 &oldID, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
434 &oldCountryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
435 &oldSender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
436 &oldLanguageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
437 &oldDateIssue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
438 &oldReferenceCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
439 &oldValue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
440 &oldPredicted, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
441 &oldValueMin, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
442 &oldValueMax, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
443 &oldDateInfo, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
444 &oldSourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
445 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
446 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
447 var newEntry bool |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
448 switch { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
449 case err == sql.ErrNoRows: |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
450 // Complete new one |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
451 newEntry = true |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
452 case err != nil: |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
453 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
454 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
455 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
456 newSender := row[fromIdx] |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
457 newLanguageCode := row[languageCodeIdx] |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
458 newCountryCode := row[countryCodeIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
459 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
460 dis, err := guessDate(row[dateIssueIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
461 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
462 return nil, fmt.Errorf("Invalid 'date_issue' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
463 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
464 newDateIssue := dis |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
465 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
466 newReferenceCode := row[referenceCodeIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
467 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
468 value, err := strconv.ParseFloat(row[valueIdx], 32) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
469 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
470 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
471 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
472 newValue := float64(convert(float32(value))) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
473 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
474 newPredicted := strings.ToLower(row[predictedIdx]) == "true" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
475 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
476 var newValueMin sql.NullFloat64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
477 if vm := row[valueMinIdx]; vm != "" { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
478 valueMin, err := strconv.ParseFloat(vm, 32) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
479 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
480 return nil, fmt.Errorf("Invalid 'value_min' line %d: %v", line, err) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
481 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
482 newValueMin = sql.NullFloat64{ |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
483 Float64: float64(convert(float32(valueMin))), |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
484 Valid: true, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
485 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
486 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
487 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
488 var newValueMax sql.NullFloat64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
489 if vm := row[valueMaxIdx]; vm != "" { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
490 valueMax, err := strconv.ParseFloat(vm, 32) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
491 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
492 return nil, fmt.Errorf("Invalid 'value_max' line %d: %v", line, err) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
493 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
494 newValueMax = sql.NullFloat64{ |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
495 Float64: float64(convert(float32(valueMax))), |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
496 Valid: true, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
497 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
498 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
499 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
500 din, err := guessDate(row[dateInfoIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
501 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
502 return nil, fmt.Errorf("Invalid 'date_info' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
503 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
504 newDateInfo := din |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
505 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
506 newSourceOrganization := row[originatorIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
507 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
508 var newID int64 |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
509 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
510 if err := insertStmt.QueryRowContext( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
511 ctx, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
512 gid.CountryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
513 gid.LoCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
514 gid.FairwaySection, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
515 gid.Orc, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
516 gid.Hectometre, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
517 md, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
518 newCountryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
519 newSender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
520 newLanguageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
521 newDateIssue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
522 newReferenceCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
523 newValue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
524 newPredicted, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
525 newValueMin, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
526 newValueMax, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
527 newDateInfo, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
528 newSourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
529 ).Scan(&newID); err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
530 return nil, err |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
531 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
532 if _, err := trackStmt.ExecContext( |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
533 ctx, importID, "waterway.gauge_measurements", newID, |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
534 ); err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
535 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
536 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
537 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
538 n := newAGMLine( |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
539 newCountryCode, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
540 newSender, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
541 newLanguageCode, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
542 newDateIssue, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
543 newReferenceCode, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
544 newValue, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
545 newPredicted, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
546 newValueMin, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
547 newValueMax, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
548 newDateInfo, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
549 newSourceOrganization, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
550 ) |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
551 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
552 ase := &agmSummaryEntry{ |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
553 FKGaugeID: *gid, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
554 MeasureDate: timetz{md}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
555 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
556 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
557 if newEntry { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
558 ase.Versions = []*agmLine{n} |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
559 } else { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
560 o := newAGMLine( |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
561 oldCountryCode, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
562 oldSender, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
563 oldLanguageCode, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
564 oldDateIssue, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
565 oldReferenceCode, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
566 oldValue, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
567 oldPredicted, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
568 oldValueMin, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
569 oldValueMax, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
570 oldDateInfo, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
571 oldSourceOrganization, |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
572 ) |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
573 // Ignore if there is no diff. |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
574 if !n.hasDiff(o) { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
575 continue |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
576 } |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
577 ase.Versions = []*agmLine{o, n} |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
578 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
579 entries = append(entries, ase) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
580 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
581 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
582 if err := tx.Commit(); err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
583 return nil, fmt.Errorf("Commit failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
584 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
585 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
586 feedback.Info("Importing approved gauge measurements took %s", |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
587 time.Since(start)) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
588 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
589 return entries, nil |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
590 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
591 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
592 func newAGMLine( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
593 countryCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
594 sender string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
595 languageCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
596 dateIssue time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
597 referenceCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
598 waterLevel float64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
599 predicted bool, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
600 valueMin sql.NullFloat64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
601 valueMax sql.NullFloat64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
602 dateInfo time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
603 sourceOrganization string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
604 ) *agmLine { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
605 nilFloat := func(v sql.NullFloat64) *float64 { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
606 var p *float64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
607 if v.Valid { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
608 p = &v.Float64 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
609 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
610 return p |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
611 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
612 return &agmLine{ |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
613 CountryCode: countryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
614 Sender: sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
615 LanguageCode: languageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
616 DateIssue: timetz{dateIssue}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
617 ReferenceCode: referenceCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
618 WaterLevel: waterLevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
619 Predicted: predicted, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
620 ValueMin: nilFloat(valueMin), |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
621 ValueMax: nilFloat(valueMax), |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
622 DateInfo: timetz{dateInfo}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
623 SourceOrganization: sourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
624 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
625 } |