Mercurial > gemma
annotate pkg/imports/agm.go @ 4177:8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Mon, 05 Aug 2019 17:16:46 +0200 |
parents | 5a650cde0574 |
children | 51e90370eced |
rev | line source |
---|---|
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
13 // * Sascha Wilde <wilde@intevation.de> |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
14 // * Tom Gottfried <tom.gottfried@intevation.de> |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
19 "bufio" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "context" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "database/sql" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
22 "encoding/csv" |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
23 "encoding/json" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
24 "fmt" |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
25 "io" |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
26 "math" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "os" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
28 "path/filepath" |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
29 "sort" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
30 "strconv" |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
31 "strings" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
32 "time" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/common" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/models" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 type ApprovedGaugeMeasurements struct { |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
39 Dir string `json:"dir"` |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
40 Originator string `json:"originator"` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 |
4146
f464cbcdf2f2
BN import: More clean-ups. Typo fixes. Code simplifications. Calling conventions ...
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4080
diff
changeset
|
43 // AGMJobKind is the unique name of an approved gauge measurements import job. |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 const AGMJobKind JobKind = "agm" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 type agmJobCreator struct{} |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 func init() { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 RegisterJobCreator(AGMJobKind, agmJobCreator{}) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
52 func (agmJobCreator) AutoAccept() bool { return false } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
53 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 func (agmJobCreator) Description() string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 return "approved gauge measurements" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2173
diff
changeset
|
58 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
60 func (agmJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
61 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
62 {"gauge_measurements"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
63 {"gauges"}, |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 const ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
68 agmStageDoneDeleteSQL = ` |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
69 DELETE FROM waterway.gauge_measurements WHERE id IN ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
70 SELECT key |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
71 FROM import.track_imports |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
72 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
73 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
74 deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
75 )` |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
76 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 agmStageDoneSQL = ` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 UPDATE waterway.gauge_measurements SET staging_done = true |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
79 WHERE id IN ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
80 SELECT key |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
81 FROM import.track_imports |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
83 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
84 NOT deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
85 )` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 func (agmJobCreator) StageDone( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 tx *sql.Tx, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 id int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 ) error { |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
93 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
94 if err == nil { |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
95 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
96 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 return err |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 // CleanUp removes the folder containing the CSV file with the |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 // the approved gauge measurements. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 func (agm *ApprovedGaugeMeasurements) CleanUp() error { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 return os.RemoveAll(agm.Dir) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
106 var guessDate = common.TimeParser([]string{ |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
107 "02.01.2006 15:04", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
108 "2006-01-02T15:04:05-07:00", |
4074
eb2f949ddfa2
Rename TimeGuesser to TimeParser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4060
diff
changeset
|
109 }).Parse |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
110 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
111 type timetz struct{ time.Time } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
112 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
113 func (ttz *timetz) MarshalJSON() ([]byte, error) { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
114 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00")) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
115 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
116 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
117 type agmLine struct { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
118 id int64 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
119 Location models.Isrs `json:"fk-gauge-id"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
120 CountryCode string `json:"country-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
121 Sender string `json:"sender"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
122 LanguageCode string `json:"language-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
123 DateIssue timetz `json:"date-issue"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
124 ReferenceCode string `json:"reference-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
125 MeasureDate timetz `json:"measure-date"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
126 WaterLevel float64 `json:"water-level"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
127 DateInfo timetz `json:"date-info"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
128 SourceOrganization string `json:"source-organization"` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
129 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
130 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
131 func (a *agmLine) hasDiff(b *agmLine) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
132 const eps = 0.00001 |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
133 return a.CountryCode != b.CountryCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
134 a.Sender != b.Sender || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
135 a.LanguageCode != b.LanguageCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
136 a.ReferenceCode != b.ReferenceCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
137 math.Abs(a.WaterLevel-b.WaterLevel) > eps || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
138 a.SourceOrganization != b.SourceOrganization |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
139 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
140 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
141 type agmSummaryEntry struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
142 FKGaugeID models.Isrs `json:"fk-gauge-id"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
143 MeasureDate timetz `json:"measure-date"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
144 Versions []*agmLine `json:"versions"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
145 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
146 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
147 const ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
148 agmSelectSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
149 SELECT |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
150 id, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
151 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
152 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
153 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
154 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
155 reference_code, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
156 measure_date, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
157 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
158 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
159 source_organization |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
160 FROM waterway.gauge_measurements |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
161 WHERE |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
162 location |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
163 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
164 AND measure_date BETWEEN $6 AND $7 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
165 AND staging_done |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
166 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
167 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
168 agmInsertSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
169 INSERT INTO waterway.gauge_measurements ( |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
170 location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
171 measure_date, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
172 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
173 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
174 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
175 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
176 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
177 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
178 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
179 source_organization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
180 staging_done |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
181 ) VALUES ( |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
182 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
183 $6, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
184 $7, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
185 $8, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
186 $9, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
187 $10, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
188 $11, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
189 $12, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
190 $13, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
191 $14, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
192 false |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
193 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
194 RETURNING id` |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
195 |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
196 agmGaugeCheckSQL = ` |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
197 SELECT EXISTS( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
198 SELECT 1 FROM waterway.gauges |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
199 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
200 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
201 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
202 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
203 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
204 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
205 headerFields := []struct { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
206 idx *int |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
207 name string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
208 }{ |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
209 {fkGaugeIDIdx, "fk_gauge_id"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
210 {measureDateIdx, "measure_date"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
211 {valueIdx, "value"}, // "water_level", |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
212 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
213 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
214 nextHeader: |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
215 for i, f := range headers { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
216 h := strings.Replace(strings.ToLower( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
217 strings.TrimSpace(f)), " ", "_", -1) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
218 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
219 for j := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
220 if headerFields[j].name == h { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
221 if *headerFields[j].idx != -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
222 return fmt.Errorf( |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
223 "there is more than one column namend '%s'", h) |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
224 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
225 *headerFields[j].idx = i |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
226 continue nextHeader |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
227 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
228 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
229 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
230 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
231 var missing []string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
232 for i := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
233 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
234 missing = append(missing, headerFields[i].name) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
235 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
236 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
237 if len(missing) > 0 { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
238 return fmt.Errorf("missing columns: %s", strings.Join(missing, ", ")) |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
239 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
240 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
241 return nil |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
242 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
243 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
244 // Do executes the actual approved gauge measurements import. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
245 func (agm *ApprovedGaugeMeasurements) Do( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
246 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
247 importID int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
248 conn *sql.Conn, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
249 feedback Feedback, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
250 ) (interface{}, error) { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
251 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
252 start := time.Now() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
253 |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
254 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv")) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
255 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
256 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
257 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
258 defer f.Close() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
259 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
260 r := csv.NewReader(bufio.NewReader(f)) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
261 r.Comma = ';' |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
262 r.ReuseRecord = true |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
263 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
264 headers, err := r.Read() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
265 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
266 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
267 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
268 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
269 var ( |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
270 fkGaugeIDIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
271 measureDateIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
272 valueIdx = -1 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
273 ) |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
274 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
275 if err := parseAGMHeaders( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
276 headers, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
277 &fkGaugeIDIdx, &measureDateIdx, &valueIdx, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
278 ); err != nil { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
279 return nil, err |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
280 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
281 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
282 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL) |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
283 if err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
284 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
285 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
286 defer gaugeCheckStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
287 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
288 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
289 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
290 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
291 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
292 defer selectStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
293 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
294 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
295 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
296 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
297 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
298 defer insertStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
299 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
300 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
301 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
302 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
303 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
304 defer trackStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
305 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
306 entries := []*agmSummaryEntry{} |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
307 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
308 checkedGauges := map[models.Isrs]bool{} |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
309 |
4171
5a650cde0574
Move warning limiter to common package to flesh up the misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4170
diff
changeset
|
310 warnLimiter := common.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100} |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
311 warn := warnLimiter.Warn |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
312 defer warnLimiter.Close() |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
313 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
314 agmLines := []*agmLine{} |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
315 ignored := 0 |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
316 mdMinMax := map[models.Isrs]*[2]time.Time{} |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
317 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
318 lines: |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
319 for line := 1; ; line++ { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
320 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
321 row, err := r.Read() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
322 switch { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
323 case err == io.EOF || len(row) == 0: |
3218
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
324 feedback.Info("Read %d entries in CSV file", line-1) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
325 if ignored > 0 { |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
326 feedback.Info("%d entries ignored", ignored) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
327 } |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
328 if ignored == line-1 { |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
329 return nil, UnchangedError("No entries imported") |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
330 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
331 break lines |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
332 case err != nil: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
333 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
334 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
335 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
336 gids := row[fkGaugeIDIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
337 gid, err := models.IsrsFromString(gids) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
338 if err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
339 return nil, fmt.Errorf("invalid ISRS code line %d: %v", line, err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
340 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
341 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
342 if exists, found := checkedGauges[*gid]; found { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
343 if !exists { |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
344 // Just ignore the line since we have already warned |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
345 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
346 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
347 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
348 } else { // not found in gauge cache |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
349 if err := gaugeCheckStmt.QueryRowContext( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
350 ctx, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
351 gid.CountryCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
352 gid.LoCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
353 gid.FairwaySection, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
354 gid.Orc, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
355 gid.Hectometre, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
356 ).Scan(&exists); err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
357 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
358 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
359 checkedGauges[*gid] = exists |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
360 if !exists { |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
361 warn("Ignoring data for unknown gauge %s", gid.String()) |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
362 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
363 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
364 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
365 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
366 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
367 md, err := guessDate(row[measureDateIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
368 if err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
369 return nil, fmt.Errorf("invalid 'measure_date' line %d: %v", line, err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
370 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
371 if v := mdMinMax[*gid]; v != nil { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
372 if md.Before(v[0]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
373 v[0] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
374 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
375 if md.After(v[1]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
376 v[1] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
377 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
378 } else { |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
379 mdMinMax[*gid] = &[2]time.Time{md, md} |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
380 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
381 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
382 newSender := agm.Originator |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
383 newCountryCode := gid.CountryCode |
4170
afabd52212d2
Moved country -> spoken language lookup table to common package to flesh out the misc package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4146
diff
changeset
|
384 newLanguageCode := common.CCtoLang[gid.CountryCode] |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
385 newDateIssue := time.Now() |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
386 newReferenceCode := "ZPG" |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
387 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
388 value, err := strconv.ParseFloat(row[valueIdx], 32) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
389 if err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
390 return nil, fmt.Errorf("invalid 'value' line %d: %v", line, err) |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
391 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
392 newValue := value |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
393 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
394 newDateInfo := newDateIssue |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
395 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
396 newSourceOrganization := newSender |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
397 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
398 agmLines = append(agmLines, newAGMLine( |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
399 *gid, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
400 newCountryCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
401 newSender, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
402 newLanguageCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
403 newDateIssue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
404 newReferenceCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
405 md, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
406 newValue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
407 newDateInfo, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
408 newSourceOrganization, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
409 )) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
410 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
411 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
412 oldGMLines := map[models.Isrs]map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
413 for gid, minMax := range mdMinMax { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
414 oldGMLines[gid], err = getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
415 ctx, selectStmt, gid, minMax[0], minMax[1]) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
416 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
417 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
418 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
419 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
420 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
421 tx, err := conn.BeginTx(ctx, nil) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
422 if err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
423 return nil, err |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
424 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
425 defer tx.Rollback() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
426 |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
427 txInsertStmt := tx.StmtContext(ctx, insertStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
428 txTrackStmt := tx.StmtContext(ctx, trackStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
429 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
430 agmLines: |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
431 for _, line := range agmLines { |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
432 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
433 var ase *agmSummaryEntry |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
434 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
435 if old := oldGMLines[line.Location]; old != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
436 ut := line.MeasureDate.Unix() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
437 if o, ok := old[ut]; ok { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
438 if !o.hasDiff(line) { // identical |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
439 // don't delete |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
440 delete(old, ut) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
441 continue agmLines |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
442 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
443 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
444 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
445 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
446 Versions: []*agmLine{o, line}, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
447 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
448 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
449 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
450 if ase == nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
451 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
452 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
453 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
454 Versions: []*agmLine{line}, |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
455 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
456 } |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
457 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
458 var newID int64 |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
459 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
460 if err := txInsertStmt.QueryRowContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
461 ctx, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
462 line.Location.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
463 line.Location.LoCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
464 line.Location.FairwaySection, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
465 line.Location.Orc, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
466 line.Location.Hectometre, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
467 line.MeasureDate.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
468 line.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
469 line.Sender, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
470 line.LanguageCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
471 line.DateIssue.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
472 line.ReferenceCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
473 line.WaterLevel, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
474 line.DateInfo.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
475 line.SourceOrganization, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
476 ).Scan(&newID); err != nil { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
477 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
478 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
479 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
480 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
481 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
482 newID, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
483 false, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
484 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
485 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
486 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
487 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
488 entries = append(entries, ase) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
489 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
490 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
491 var removed int |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
492 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
493 // Issue deletes |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
494 for _, old := range oldGMLines { |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
495 removed += len(old) |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
496 for _, line := range old { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
497 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
498 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
499 line.id, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
500 true, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
501 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
502 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
503 } |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
504 entries = append(entries, &agmSummaryEntry{ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
505 FKGaugeID: line.Location, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
506 MeasureDate: line.MeasureDate, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
507 Versions: []*agmLine{line, nil}, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
508 }) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
509 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
510 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
511 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
512 feedback.Info("Measurements to update/insert: %d", len(entries)) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
513 feedback.Info("Measurements to delete: %d", removed) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
514 |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
515 if len(entries) == 0 && removed == 0 { |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
516 return nil, UnchangedError("No changes from AGM import") |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
517 } |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
518 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
519 if err = tx.Commit(); err != nil { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4171
diff
changeset
|
520 return nil, fmt.Errorf("commit failed: %v", err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
521 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
522 |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
523 // Sort here to mix the deletes right beside the matching inserts/updates. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
524 // This also makes the output deterministic. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
525 sort.Slice(entries, func(i, j int) bool { |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
526 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
527 }) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
528 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
529 feedback.Info("Imported %d entries with changes", len(entries)) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
530 feedback.Info("Importing approved gauge measurements took %s", |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
531 time.Since(start)) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
532 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
533 return entries, nil |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
534 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
535 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
536 func getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
537 ctx context.Context, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
538 stmt *sql.Stmt, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
539 location models.Isrs, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
540 from time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
541 to time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
542 ) (map[int64]*agmLine, error) { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
543 var ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
544 oldID int64 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
545 oldCountryCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
546 oldSender string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
547 oldLanguageCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
548 oldDateIssue time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
549 oldReferenceCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
550 oldMeasureDate time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
551 oldValue float64 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
552 oldDateInfo time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
553 oldSourceOrganization string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
554 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
555 gmLines := map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
556 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
557 gms, err := stmt.QueryContext( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
558 ctx, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
559 location.CountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
560 location.LoCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
561 location.FairwaySection, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
562 location.Orc, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
563 location.Hectometre, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
564 from, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
565 to, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
566 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
567 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
568 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
569 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
570 defer gms.Close() |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
571 for gms.Next() { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
572 if err = gms.Scan( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
573 &oldID, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
574 &oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
575 &oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
576 &oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
577 &oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
578 &oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
579 &oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
580 &oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
581 &oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
582 &oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
583 ); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
584 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
585 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
586 line := newAGMLine( |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
587 location, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
588 oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
589 oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
590 oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
591 oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
592 oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
593 oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
594 oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
595 oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
596 oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
597 ) |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
598 line.id = oldID |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
599 gmLines[oldMeasureDate.Unix()] = line |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
600 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
601 if err = gms.Err(); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
602 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
603 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
604 return gmLines, nil |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
605 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
606 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
607 func newAGMLine( |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
608 location models.Isrs, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
609 countryCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
610 sender string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
611 languageCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
612 dateIssue time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
613 referenceCode string, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
614 measureDate time.Time, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
615 waterLevel float64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
616 dateInfo time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
617 sourceOrganization string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
618 ) *agmLine { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
619 return &agmLine{ |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
620 Location: location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
621 CountryCode: countryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
622 Sender: sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
623 LanguageCode: languageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
624 DateIssue: timetz{dateIssue}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
625 ReferenceCode: referenceCode, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
626 MeasureDate: timetz{measureDate}, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
627 WaterLevel: waterLevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
628 DateInfo: timetz{dateInfo}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
629 SourceOrganization: sourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
630 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
631 } |