Mercurial > gemma
annotate pkg/imports/agm.go @ 4087:1562a5fb36dc
Handle Nash-Sutcliff more tolerant for NaN/Inf values.
Due to limitation of JSON we encode these values as 0 and +/- MaxFloat64.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Thu, 25 Jul 2019 17:15:00 +0200 |
parents | cb74aa69954e |
children | af2b20d6c921 |
rev | line source |
---|---|
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
13 // * Sascha Wilde <wilde@intevation.de> |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
14 // * Tom Gottfried <tom.gottfried@intevation.de> |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
19 "bufio" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "context" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "database/sql" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
22 "encoding/csv" |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
23 "encoding/json" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
24 "fmt" |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
25 "io" |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
26 "math" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "os" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
28 "path/filepath" |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
29 "sort" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
30 "strconv" |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
31 "strings" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
32 "time" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/common" |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/misc" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
36 "gemma.intevation.de/gemma/pkg/models" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
39 type ApprovedGaugeMeasurements struct { |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
40 Dir string `json:"dir"` |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
41 Originator string `json:"originator"` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 // GMAPJobKind is the unique name of an approved gauge measurements import job. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 const AGMJobKind JobKind = "agm" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 type agmJobCreator struct{} |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 func init() { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 RegisterJobCreator(AGMJobKind, agmJobCreator{}) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
53 func (agmJobCreator) AutoAccept() bool { return false } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
54 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 func (agmJobCreator) Description() string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 return "approved gauge measurements" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2173
diff
changeset
|
59 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
61 func (agmJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
62 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
63 {"gauge_measurements"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
64 {"gauges"}, |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 const ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
69 agmStageDoneDeleteSQL = ` |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
70 DELETE FROM waterway.gauge_measurements WHERE id IN ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
71 SELECT key |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
72 FROM import.track_imports |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
73 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
74 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
75 deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
76 )` |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
77 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 agmStageDoneSQL = ` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 UPDATE waterway.gauge_measurements SET staging_done = true |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
80 WHERE id IN ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
81 SELECT key |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
82 FROM import.track_imports |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
84 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
85 NOT deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
86 )` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 func (agmJobCreator) StageDone( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 tx *sql.Tx, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 id int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 ) error { |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
94 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
95 if err == nil { |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
96 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
97 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 return err |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 // CleanUp removes the folder containing the CSV file with the |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 // the approved gauge measurements. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 func (agm *ApprovedGaugeMeasurements) CleanUp() error { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 return os.RemoveAll(agm.Dir) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
107 var guessDate = common.TimeParser([]string{ |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
108 "02.01.2006 15:04", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
109 "2006-01-02T15:04:05-07:00", |
4074
eb2f949ddfa2
Rename TimeGuesser to TimeParser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4060
diff
changeset
|
110 }).Parse |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
111 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
112 type timetz struct{ time.Time } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
113 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
114 func (ttz *timetz) MarshalJSON() ([]byte, error) { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
115 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00")) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
116 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
117 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
118 type agmLine struct { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
119 id int64 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
120 Location models.Isrs `json:"fk-gauge-id"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
121 CountryCode string `json:"country-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
122 Sender string `json:"sender"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
123 LanguageCode string `json:"language-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
124 DateIssue timetz `json:"date-issue"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
125 ReferenceCode string `json:"reference-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
126 MeasureDate timetz `json:"measure-date"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
127 WaterLevel float64 `json:"water-level"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
128 DateInfo timetz `json:"date-info"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
129 SourceOrganization string `json:"source-organization"` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
130 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
131 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
132 func (a *agmLine) hasDiff(b *agmLine) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
133 const eps = 0.00001 |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
134 return a.CountryCode != b.CountryCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
135 a.Sender != b.Sender || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
136 a.LanguageCode != b.LanguageCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
137 a.ReferenceCode != b.ReferenceCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
138 math.Abs(a.WaterLevel-b.WaterLevel) > eps || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
139 a.SourceOrganization != b.SourceOrganization |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
140 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
141 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
142 type agmSummaryEntry struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
143 FKGaugeID models.Isrs `json:"fk-gauge-id"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
144 MeasureDate timetz `json:"measure-date"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
145 Versions []*agmLine `json:"versions"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
146 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
147 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
148 const ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
149 agmSelectSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
150 SELECT |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
151 id, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
152 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
153 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
154 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
155 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
156 reference_code, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
157 measure_date, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
158 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
159 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
160 source_organization |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
161 FROM waterway.gauge_measurements |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
162 WHERE |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
163 location |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
164 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
165 AND measure_date BETWEEN $6 AND $7 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
166 AND staging_done |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
167 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
168 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
169 agmInsertSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
170 INSERT INTO waterway.gauge_measurements ( |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
171 location, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
172 validity, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
173 measure_date, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
174 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
175 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
176 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
177 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
178 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
179 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
180 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
181 source_organization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
182 staging_done |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
183 ) VALUES ( |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
184 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
3533
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
185 COALESCE( |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
186 (SELECT validity FROM waterway.gauges |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
187 WHERE location |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
188 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
189 AND validity @> CAST($6 AS timestamp with time zone)), |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
190 tstzrange(NULL, NULL)), |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
191 $6, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
192 $7, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
193 $8, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
194 $9, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
195 $10, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
196 $11, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
197 $12, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
198 $13, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
199 $14, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
200 false |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
201 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
202 RETURNING id` |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
203 |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
204 agmGaugeCheckSQL = ` |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
205 SELECT EXISTS( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
206 SELECT 1 FROM waterway.gauges |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
207 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
208 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
209 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
210 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
211 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
212 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
213 headerFields := []struct { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
214 idx *int |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
215 name string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
216 }{ |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
217 {fkGaugeIDIdx, "fk_gauge_id"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
218 {measureDateIdx, "measure_date"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
219 {valueIdx, "value"}, // "water_level", |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
220 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
221 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
222 nextHeader: |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
223 for i, f := range headers { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
224 h := strings.Replace(strings.ToLower( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
225 strings.TrimSpace(f)), " ", "_", -1) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
226 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
227 for j := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
228 if headerFields[j].name == h { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
229 if *headerFields[j].idx != -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
230 return fmt.Errorf( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
231 "There is more than one column namend '%s'", h) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
232 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
233 *headerFields[j].idx = i |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
234 continue nextHeader |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
235 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
236 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
237 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
238 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
239 var missing []string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
240 for i := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
241 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
242 missing = append(missing, headerFields[i].name) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
243 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
244 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
245 if len(missing) > 0 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
246 return fmt.Errorf("Missing columns: %s", strings.Join(missing, ", ")) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
247 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
248 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
249 return nil |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
250 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
251 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
252 // Do executes the actual approved gauge measurements import. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
253 func (agm *ApprovedGaugeMeasurements) Do( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
254 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
255 importID int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
256 conn *sql.Conn, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
257 feedback Feedback, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
258 ) (interface{}, error) { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
259 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
260 start := time.Now() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
261 |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
262 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv")) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
263 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
264 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
265 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
266 defer f.Close() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
267 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
268 r := csv.NewReader(bufio.NewReader(f)) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
269 r.Comma = ';' |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
270 r.ReuseRecord = true |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
271 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
272 headers, err := r.Read() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
273 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
274 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
275 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
276 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
277 var ( |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
278 fkGaugeIDIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
279 measureDateIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
280 valueIdx = -1 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
281 ) |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
282 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
283 if err := parseAGMHeaders( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
284 headers, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
285 &fkGaugeIDIdx, &measureDateIdx, &valueIdx, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
286 ); err != nil { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
287 return nil, err |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
288 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
289 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
290 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL) |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
291 if err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
292 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
293 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
294 defer gaugeCheckStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
295 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
296 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
297 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
298 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
299 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
300 defer selectStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
301 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
302 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
303 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
304 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
305 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
306 defer insertStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
307 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
308 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
309 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
310 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
311 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
312 defer trackStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
313 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
314 entries := []*agmSummaryEntry{} |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
315 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
316 checkedGauges := map[models.Isrs]bool{} |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
317 |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
318 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100} |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
319 warn := warnLimiter.Warn |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
320 defer warnLimiter.Close() |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
321 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
322 agmLines := []*agmLine{} |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
323 ignored := 0 |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
324 mdMinMax := map[models.Isrs]*[2]time.Time{} |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
325 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
326 lines: |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
327 for line := 1; ; line++ { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
328 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
329 row, err := r.Read() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
330 switch { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
331 case err == io.EOF || len(row) == 0: |
3218
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
332 feedback.Info("Read %d entries in CSV file", line-1) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
333 if ignored > 0 { |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
334 feedback.Info("%d entries ignored", ignored) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
335 } |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
336 if ignored == line-1 { |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
337 return nil, UnchangedError("No entries imported") |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
338 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
339 break lines |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
340 case err != nil: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
341 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
342 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
343 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
344 gids := row[fkGaugeIDIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
345 gid, err := models.IsrsFromString(gids) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
346 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
347 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
348 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
349 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
350 if exists, found := checkedGauges[*gid]; found { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
351 if !exists { |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
352 // Just ignore the line since we have already warned |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
353 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
354 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
355 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
356 } else { // not found in gauge cache |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
357 if err := gaugeCheckStmt.QueryRowContext( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
358 ctx, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
359 gid.CountryCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
360 gid.LoCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
361 gid.FairwaySection, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
362 gid.Orc, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
363 gid.Hectometre, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
364 ).Scan(&exists); err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
365 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
366 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
367 checkedGauges[*gid] = exists |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
368 if !exists { |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
369 warn("Ignoring data for unknown gauge %s", gid.String()) |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
370 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
371 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
372 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
373 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
374 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
375 md, err := guessDate(row[measureDateIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
376 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
377 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
378 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
379 if v := mdMinMax[*gid]; v != nil { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
380 if md.Before(v[0]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
381 v[0] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
382 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
383 if md.After(v[1]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
384 v[1] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
385 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
386 } else { |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
387 mdMinMax[*gid] = &[2]time.Time{md, md} |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
388 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
389 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
390 newSender := agm.Originator |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
391 newCountryCode := gid.CountryCode |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
392 newLanguageCode := misc.CCtoLang[gid.CountryCode] |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
393 newDateIssue := time.Now() |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
394 newReferenceCode := "ZPG" |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
395 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
396 value, err := strconv.ParseFloat(row[valueIdx], 32) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
397 if err != nil { |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
398 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
399 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
400 newValue := value |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
401 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
402 newDateInfo := newDateIssue |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
403 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
404 newSourceOrganization := newSender |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
405 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
406 agmLines = append(agmLines, newAGMLine( |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
407 *gid, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
408 newCountryCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
409 newSender, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
410 newLanguageCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
411 newDateIssue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
412 newReferenceCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
413 md, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
414 newValue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
415 newDateInfo, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
416 newSourceOrganization, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
417 )) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
418 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
419 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
420 oldGMLines := map[models.Isrs]map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
421 for gid, minMax := range mdMinMax { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
422 oldGMLines[gid], err = getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
423 ctx, selectStmt, gid, minMax[0], minMax[1]) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
424 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
425 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
426 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
427 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
428 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
429 tx, err := conn.BeginTx(ctx, nil) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
430 if err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
431 return nil, err |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
432 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
433 defer tx.Rollback() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
434 |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
435 txInsertStmt := tx.StmtContext(ctx, insertStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
436 txTrackStmt := tx.StmtContext(ctx, trackStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
437 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
438 agmLines: |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
439 for _, line := range agmLines { |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
440 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
441 var ase *agmSummaryEntry |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
442 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
443 if old := oldGMLines[line.Location]; old != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
444 ut := line.MeasureDate.Unix() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
445 if o, ok := old[ut]; ok { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
446 if !o.hasDiff(line) { // identical |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
447 // don't delete |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
448 delete(old, ut) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
449 continue agmLines |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
450 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
451 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
452 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
453 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
454 Versions: []*agmLine{o, line}, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
455 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
456 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
457 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
458 if ase == nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
459 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
460 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
461 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
462 Versions: []*agmLine{line}, |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
463 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
464 } |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
465 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
466 var newID int64 |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
467 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
468 if err := txInsertStmt.QueryRowContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
469 ctx, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
470 line.Location.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
471 line.Location.LoCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
472 line.Location.FairwaySection, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
473 line.Location.Orc, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
474 line.Location.Hectometre, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
475 line.MeasureDate.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
476 line.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
477 line.Sender, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
478 line.LanguageCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
479 line.DateIssue.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
480 line.ReferenceCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
481 line.WaterLevel, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
482 line.DateInfo.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
483 line.SourceOrganization, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
484 ).Scan(&newID); err != nil { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
485 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
486 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
487 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
488 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
489 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
490 newID, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
491 false, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
492 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
493 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
494 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
495 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
496 entries = append(entries, ase) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
497 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
498 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
499 var removed int |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
500 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
501 // Issue deletes |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
502 for _, old := range oldGMLines { |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
503 removed += len(old) |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
504 for _, line := range old { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
505 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
506 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
507 line.id, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
508 true, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
509 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
510 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
511 } |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
512 entries = append(entries, &agmSummaryEntry{ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
513 FKGaugeID: line.Location, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
514 MeasureDate: line.MeasureDate, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
515 Versions: []*agmLine{line, nil}, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
516 }) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
517 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
518 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
519 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
520 feedback.Info("Measurements to update/insert: %d", len(entries)) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
521 feedback.Info("Measurements to delete: %d", removed) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
522 |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
523 if len(entries) == 0 && removed == 0 { |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
524 return nil, UnchangedError("No changes from AGM import") |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
525 } |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
526 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
527 if err = tx.Commit(); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
528 return nil, fmt.Errorf("Commit failed: %v", err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
529 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
530 |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
531 // Sort here to mix the deletes right beside the matching inserts/updates. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
532 // This also makes the output deterministic. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
533 sort.Slice(entries, func(i, j int) bool { |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
534 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
535 }) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
536 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
537 feedback.Info("Imported %d entries with changes", len(entries)) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
538 feedback.Info("Importing approved gauge measurements took %s", |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
539 time.Since(start)) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
540 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
541 return entries, nil |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
542 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
543 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
544 func getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
545 ctx context.Context, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
546 stmt *sql.Stmt, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
547 location models.Isrs, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
548 from time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
549 to time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
550 ) (map[int64]*agmLine, error) { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
551 var ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
552 oldID int64 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
553 oldCountryCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
554 oldSender string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
555 oldLanguageCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
556 oldDateIssue time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
557 oldReferenceCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
558 oldMeasureDate time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
559 oldValue float64 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
560 oldDateInfo time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
561 oldSourceOrganization string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
562 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
563 gmLines := map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
564 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
565 gms, err := stmt.QueryContext( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
566 ctx, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
567 location.CountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
568 location.LoCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
569 location.FairwaySection, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
570 location.Orc, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
571 location.Hectometre, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
572 from, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
573 to, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
574 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
575 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
576 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
577 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
578 defer gms.Close() |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
579 for gms.Next() { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
580 if err = gms.Scan( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
581 &oldID, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
582 &oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
583 &oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
584 &oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
585 &oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
586 &oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
587 &oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
588 &oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
589 &oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
590 &oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
591 ); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
592 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
593 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
594 line := newAGMLine( |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
595 location, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
596 oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
597 oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
598 oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
599 oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
600 oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
601 oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
602 oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
603 oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
604 oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
605 ) |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
606 line.id = oldID |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
607 gmLines[oldMeasureDate.Unix()] = line |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
608 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
609 if err = gms.Err(); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
610 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
611 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
612 return gmLines, nil |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
613 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
614 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
615 func newAGMLine( |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
616 location models.Isrs, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
617 countryCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
618 sender string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
619 languageCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
620 dateIssue time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
621 referenceCode string, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
622 measureDate time.Time, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
623 waterLevel float64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
624 dateInfo time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
625 sourceOrganization string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
626 ) *agmLine { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
627 return &agmLine{ |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
628 Location: location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
629 CountryCode: countryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
630 Sender: sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
631 LanguageCode: languageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
632 DateIssue: timetz{dateIssue}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
633 ReferenceCode: referenceCode, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
634 MeasureDate: timetz{measureDate}, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
635 WaterLevel: waterLevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
636 DateInfo: timetz{dateInfo}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
637 SourceOrganization: sourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
638 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
639 } |