Mercurial > gemma
annotate pkg/imports/agm.go @ 5532:7e21a9e6353d aggregate-gm-import-logging
Cleaned up and simplified code.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Mon, 25 Oct 2021 00:37:39 +0200 |
parents | 56c589f7435d |
children | 6270951dda28 |
rev | line source |
---|---|
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
13 // * Sascha Wilde <wilde@intevation.de> |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
14 // * Tom Gottfried <tom.gottfried@intevation.de> |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
19 "bufio" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "context" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "database/sql" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
22 "encoding/csv" |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
23 "encoding/json" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
24 "fmt" |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
25 "io" |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
26 "math" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "os" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
28 "path/filepath" |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
29 "sort" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
30 "strconv" |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
31 "strings" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
32 "time" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/common" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/models" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
4178
51e90370eced
A few more doc strings to make 'golint' a little bit more happy with the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
38 // ApprovedGaugeMeasurements is a Job to import |
51e90370eced
A few more doc strings to make 'golint' a little bit more happy with the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
39 // approved gauge measurements from a CVS file |
51e90370eced
A few more doc strings to make 'golint' a little bit more happy with the imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
40 // into the database. |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 type ApprovedGaugeMeasurements struct { |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
42 Dir string `json:"dir"` |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
43 Originator string `json:"originator"` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4080
diff
changeset
|
46 // AGMJobKind is the unique name of an approved gauge measurements import job. |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 const AGMJobKind JobKind = "agm" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 type agmJobCreator struct{} |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 func init() { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 RegisterJobCreator(AGMJobKind, agmJobCreator{}) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
55 func (agmJobCreator) AutoAccept() bool { return false } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
56 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 func (agmJobCreator) Description() string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 return "approved gauge measurements" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2173
diff
changeset
|
61 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
63 func (agmJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
64 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
65 {"gauge_measurements"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
66 {"gauges"}, |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 const ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
71 agmStageDoneDeleteSQL = ` |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
72 DELETE FROM waterway.gauge_measurements WHERE id IN ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
73 SELECT key |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
74 FROM import.track_imports |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
75 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
76 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
77 deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
78 )` |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
79 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 agmStageDoneSQL = ` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 UPDATE waterway.gauge_measurements SET staging_done = true |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
82 WHERE id IN ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
83 SELECT key |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
84 FROM import.track_imports |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
86 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
87 NOT deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
88 )` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 |
5035
56c589f7435d
Enhance comments on implementations of StageDone()
Tom Gottfried <tom@intevation.de>
parents:
5034
diff
changeset
|
91 // StageDone replaces gauge measurements with those in the staging area |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 func (agmJobCreator) StageDone( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 tx *sql.Tx, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 id int64, |
5034
59a99655f34d
Added feedback support for StageDone.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4178
diff
changeset
|
96 _ Feedback, |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 ) error { |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
98 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
99 if err == nil { |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
100 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
101 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 return err |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 // CleanUp removes the folder containing the CSV file with the |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 // the approved gauge measurements. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 func (agm *ApprovedGaugeMeasurements) CleanUp() error { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
108 return os.RemoveAll(agm.Dir) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
109 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
110 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
111 var guessDate = common.TimeParser([]string{ |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
112 "02.01.2006 15:04", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
113 "2006-01-02T15:04:05-07:00", |
4074
eb2f949ddfa2
Rename TimeGuesser to TimeParser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4060
diff
changeset
|
114 }).Parse |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
115 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
116 type timetz struct{ time.Time } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
117 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
118 func (ttz *timetz) MarshalJSON() ([]byte, error) { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
119 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00")) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
120 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
121 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
122 type agmLine struct { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
123 id int64 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
124 Location models.Isrs `json:"fk-gauge-id"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
125 CountryCode string `json:"country-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
126 Sender string `json:"sender"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
127 LanguageCode string `json:"language-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
128 DateIssue timetz `json:"date-issue"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
129 ReferenceCode string `json:"reference-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
130 MeasureDate timetz `json:"measure-date"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
131 WaterLevel float64 `json:"water-level"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
132 DateInfo timetz `json:"date-info"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
133 SourceOrganization string `json:"source-organization"` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
134 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
135 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
136 func (a *agmLine) hasDiff(b *agmLine) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
137 const eps = 0.00001 |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
138 return a.CountryCode != b.CountryCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
139 a.Sender != b.Sender || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
140 a.LanguageCode != b.LanguageCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
141 a.ReferenceCode != b.ReferenceCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
142 math.Abs(a.WaterLevel-b.WaterLevel) > eps || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
143 a.SourceOrganization != b.SourceOrganization |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
144 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
145 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
146 type agmSummaryEntry struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
147 FKGaugeID models.Isrs `json:"fk-gauge-id"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
148 MeasureDate timetz `json:"measure-date"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
149 Versions []*agmLine `json:"versions"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
150 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
151 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
152 const ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
153 agmSelectSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
154 SELECT |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
155 id, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
156 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
157 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
158 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
159 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
160 reference_code, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
161 measure_date, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
162 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
163 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
164 source_organization |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
165 FROM waterway.gauge_measurements |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
166 WHERE |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
167 location |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
168 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
169 AND measure_date BETWEEN $6 AND $7 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
170 AND staging_done |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
171 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
172 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
173 agmInsertSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
174 INSERT INTO waterway.gauge_measurements ( |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
175 location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
176 measure_date, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
177 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
178 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
179 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
180 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
181 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
182 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
183 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
184 source_organization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
185 staging_done |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
186 ) VALUES ( |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
187 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
188 $6, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
189 $7, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
190 $8, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
191 $9, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
192 $10, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
193 $11, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
194 $12, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
195 $13, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
196 $14, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
197 false |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
198 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
199 RETURNING id` |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
200 |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
201 agmGaugeCheckSQL = ` |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
202 SELECT EXISTS( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
203 SELECT 1 FROM waterway.gauges |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
204 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
205 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
206 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
207 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
208 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
209 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
210 headerFields := []struct { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
211 idx *int |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
212 name string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
213 }{ |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
214 {fkGaugeIDIdx, "fk_gauge_id"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
215 {measureDateIdx, "measure_date"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
216 {valueIdx, "value"}, // "water_level", |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
217 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
218 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
219 nextHeader: |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
220 for i, f := range headers { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
221 h := strings.Replace(strings.ToLower( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
222 strings.TrimSpace(f)), " ", "_", -1) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
223 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
224 for j := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
225 if headerFields[j].name == h { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
226 if *headerFields[j].idx != -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
227 return fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
228 "there is more than one column namend '%s'", h) |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
229 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
230 *headerFields[j].idx = i |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
231 continue nextHeader |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
232 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
233 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
234 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
235 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
236 var missing []string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
237 for i := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
238 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
239 missing = append(missing, headerFields[i].name) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
240 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
241 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
242 if len(missing) > 0 { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
243 return fmt.Errorf("missing columns: %s", strings.Join(missing, ", ")) |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
244 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
245 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
246 return nil |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
247 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
248 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
249 // Do executes the actual approved gauge measurements import. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
250 func (agm *ApprovedGaugeMeasurements) Do( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
251 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
252 importID int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
253 conn *sql.Conn, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
254 feedback Feedback, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
255 ) (interface{}, error) { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
256 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
257 start := time.Now() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
258 |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
259 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv")) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
260 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
261 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
262 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
263 defer f.Close() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
264 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
265 r := csv.NewReader(bufio.NewReader(f)) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
266 r.Comma = ';' |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
267 r.ReuseRecord = true |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
268 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
269 headers, err := r.Read() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
270 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
271 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
272 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
273 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
274 var ( |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
275 fkGaugeIDIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
276 measureDateIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
277 valueIdx = -1 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
278 ) |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
279 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
280 if err := parseAGMHeaders( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
281 headers, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
282 &fkGaugeIDIdx, &measureDateIdx, &valueIdx, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
283 ); err != nil { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
284 return nil, err |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
285 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
286 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
287 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL) |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
288 if err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
289 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
290 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
291 defer gaugeCheckStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
292 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
293 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
294 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
295 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
296 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
297 defer selectStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
298 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
299 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
300 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
301 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
302 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
303 defer insertStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
304 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
305 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
306 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
307 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
308 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
309 defer trackStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
310 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
311 entries := []*agmSummaryEntry{} |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
312 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
313 checkedGauges := map[models.Isrs]bool{} |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
314 |
4171
5a650cde0574
Move warning limiter to common package to flesh up the misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4170
diff
changeset
|
315 warnLimiter := common.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100} |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
316 warn := warnLimiter.Warn |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
317 defer warnLimiter.Close() |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
318 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
319 agmLines := []*agmLine{} |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
320 ignored := 0 |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
321 mdMinMax := map[models.Isrs]*[2]time.Time{} |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
322 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
323 lines: |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
324 for line := 1; ; line++ { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
325 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
326 row, err := r.Read() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
327 switch { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
328 case err == io.EOF || len(row) == 0: |
3218
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
329 feedback.Info("Read %d entries in CSV file", line-1) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
330 if ignored > 0 { |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
331 feedback.Info("%d entries ignored", ignored) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
332 } |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
333 if ignored == line-1 { |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
334 return nil, UnchangedError("No entries imported") |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
335 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
336 break lines |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
337 case err != nil: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
338 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
339 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
340 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
341 gids := row[fkGaugeIDIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
342 gid, err := models.IsrsFromString(gids) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
343 if err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
344 return nil, fmt.Errorf("invalid ISRS code line %d: %v", line, err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
345 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
346 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
347 if exists, found := checkedGauges[*gid]; found { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
348 if !exists { |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
349 // Just ignore the line since we have already warned |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
350 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
351 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
352 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
353 } else { // not found in gauge cache |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
354 if err := gaugeCheckStmt.QueryRowContext( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
355 ctx, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
356 gid.CountryCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
357 gid.LoCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
358 gid.FairwaySection, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
359 gid.Orc, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
360 gid.Hectometre, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
361 ).Scan(&exists); err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
362 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
363 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
364 checkedGauges[*gid] = exists |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
365 if !exists { |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
366 warn("Ignoring data for unknown gauge %s", gid.String()) |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
367 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
368 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
369 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
370 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
371 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
372 md, err := guessDate(row[measureDateIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
373 if err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
374 return nil, fmt.Errorf("invalid 'measure_date' line %d: %v", line, err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
375 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
376 if v := mdMinMax[*gid]; v != nil { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
377 if md.Before(v[0]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
378 v[0] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
379 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
380 if md.After(v[1]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
381 v[1] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
382 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
383 } else { |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
384 mdMinMax[*gid] = &[2]time.Time{md, md} |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
385 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
386 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
387 newSender := agm.Originator |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
388 newCountryCode := gid.CountryCode |
4170
afabd52212d2
Moved country -> spoken language lookup table to common package to flesh out the misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4146
diff
changeset
|
389 newLanguageCode := common.CCtoLang[gid.CountryCode] |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
390 newDateIssue := time.Now() |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
391 newReferenceCode := "ZPG" |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
392 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
393 value, err := strconv.ParseFloat(row[valueIdx], 32) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
394 if err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
395 return nil, fmt.Errorf("invalid 'value' line %d: %v", line, err) |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
396 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
397 newValue := value |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
398 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
399 newDateInfo := newDateIssue |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
400 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
401 newSourceOrganization := newSender |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
402 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
403 agmLines = append(agmLines, newAGMLine( |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
404 *gid, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
405 newCountryCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
406 newSender, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
407 newLanguageCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
408 newDateIssue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
409 newReferenceCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
410 md, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
411 newValue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
412 newDateInfo, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
413 newSourceOrganization, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
414 )) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
415 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
416 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
417 oldGMLines := map[models.Isrs]map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
418 for gid, minMax := range mdMinMax { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
419 oldGMLines[gid], err = getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
420 ctx, selectStmt, gid, minMax[0], minMax[1]) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
421 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
422 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
423 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
424 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
425 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
426 tx, err := conn.BeginTx(ctx, nil) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
427 if err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
428 return nil, err |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
429 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
430 defer tx.Rollback() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
431 |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
432 txInsertStmt := tx.StmtContext(ctx, insertStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
433 txTrackStmt := tx.StmtContext(ctx, trackStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
434 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
435 agmLines: |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
436 for _, line := range agmLines { |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
437 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
438 var ase *agmSummaryEntry |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
439 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
440 if old := oldGMLines[line.Location]; old != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
441 ut := line.MeasureDate.Unix() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
442 if o, ok := old[ut]; ok { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
443 if !o.hasDiff(line) { // identical |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
444 // don't delete |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
445 delete(old, ut) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
446 continue agmLines |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
447 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
448 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
449 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
450 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
451 Versions: []*agmLine{o, line}, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
452 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
453 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
454 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
455 if ase == nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
456 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
457 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
458 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
459 Versions: []*agmLine{line}, |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
460 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
461 } |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
462 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
463 var newID int64 |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
464 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
465 if err := txInsertStmt.QueryRowContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
466 ctx, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
467 line.Location.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
468 line.Location.LoCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
469 line.Location.FairwaySection, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
470 line.Location.Orc, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
471 line.Location.Hectometre, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
472 line.MeasureDate.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
473 line.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
474 line.Sender, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
475 line.LanguageCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
476 line.DateIssue.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
477 line.ReferenceCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
478 line.WaterLevel, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
479 line.DateInfo.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
480 line.SourceOrganization, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
481 ).Scan(&newID); err != nil { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
482 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
483 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
484 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
485 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
486 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
487 newID, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
488 false, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
489 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
490 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
491 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
492 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
493 entries = append(entries, ase) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
494 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
495 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
496 var removed int |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
497 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
498 // Issue deletes |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
499 for _, old := range oldGMLines { |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
500 removed += len(old) |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
501 for _, line := range old { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
502 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
503 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
504 line.id, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
505 true, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
506 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
507 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
508 } |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
509 entries = append(entries, &agmSummaryEntry{ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
510 FKGaugeID: line.Location, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
511 MeasureDate: line.MeasureDate, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
512 Versions: []*agmLine{line, nil}, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
513 }) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
514 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
515 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
516 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
517 feedback.Info("Measurements to update/insert: %d", len(entries)) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
518 feedback.Info("Measurements to delete: %d", removed) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
519 |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
520 if len(entries) == 0 && removed == 0 { |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
521 return nil, UnchangedError("No changes from AGM import") |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
522 } |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
523 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
524 if err = tx.Commit(); err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
525 return nil, fmt.Errorf("commit failed: %v", err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
526 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
527 |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
528 // Sort here to mix the deletes right beside the matching inserts/updates. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
529 // This also makes the output deterministic. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
530 sort.Slice(entries, func(i, j int) bool { |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
531 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
532 }) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
533 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
534 feedback.Info("Imported %d entries with changes", len(entries)) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
535 feedback.Info("Importing approved gauge measurements took %s", |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
536 time.Since(start)) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
537 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
538 return entries, nil |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
539 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
540 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
541 func getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
542 ctx context.Context, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
543 stmt *sql.Stmt, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
544 location models.Isrs, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
545 from time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
546 to time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
547 ) (map[int64]*agmLine, error) { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
548 var ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
549 oldID int64 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
550 oldCountryCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
551 oldSender string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
552 oldLanguageCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
553 oldDateIssue time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
554 oldReferenceCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
555 oldMeasureDate time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
556 oldValue float64 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
557 oldDateInfo time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
558 oldSourceOrganization string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
559 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
560 gmLines := map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
561 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
562 gms, err := stmt.QueryContext( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
563 ctx, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
564 location.CountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
565 location.LoCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
566 location.FairwaySection, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
567 location.Orc, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
568 location.Hectometre, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
569 from, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
570 to, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
571 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
572 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
573 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
574 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
575 defer gms.Close() |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
576 for gms.Next() { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
577 if err = gms.Scan( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
578 &oldID, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
579 &oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
580 &oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
581 &oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
582 &oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
583 &oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
584 &oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
585 &oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
586 &oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
587 &oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
588 ); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
589 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
590 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
591 line := newAGMLine( |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
592 location, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
593 oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
594 oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
595 oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
596 oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
597 oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
598 oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
599 oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
600 oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
601 oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
602 ) |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
603 line.id = oldID |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
604 gmLines[oldMeasureDate.Unix()] = line |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
605 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
606 if err = gms.Err(); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
607 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
608 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
609 return gmLines, nil |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
610 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
611 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
612 func newAGMLine( |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
613 location models.Isrs, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
614 countryCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
615 sender string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
616 languageCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
617 dateIssue time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
618 referenceCode string, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
619 measureDate time.Time, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
620 waterLevel float64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
621 dateInfo time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
622 sourceOrganization string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
623 ) *agmLine { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
624 return &agmLine{ |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
625 Location: location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
626 CountryCode: countryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
627 Sender: sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
628 LanguageCode: languageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
629 DateIssue: timetz{dateIssue}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
630 ReferenceCode: referenceCode, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
631 MeasureDate: timetz{measureDate}, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
632 WaterLevel: waterLevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
633 DateInfo: timetz{dateInfo}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
634 SourceOrganization: sourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
635 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
636 } |