Mercurial > gemma
annotate pkg/imports/agm.go @ 4120:ad0c373dff6a request_hist_bns
Merged default
author | Sascha Wilde <wilde@intevation.de> |
---|---|
date | Wed, 31 Jul 2019 19:08:53 +0200 |
parents | af2b20d6c921 |
children | f464cbcdf2f2 |
rev | line source |
---|---|
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
13 // * Sascha Wilde <wilde@intevation.de> |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
14 // * Tom Gottfried <tom.gottfried@intevation.de> |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
19 "bufio" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "context" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "database/sql" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
22 "encoding/csv" |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
23 "encoding/json" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
24 "fmt" |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
25 "io" |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
26 "math" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "os" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
28 "path/filepath" |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
29 "sort" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
30 "strconv" |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
31 "strings" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
32 "time" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/common" |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/misc" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
36 "gemma.intevation.de/gemma/pkg/models" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
39 type ApprovedGaugeMeasurements struct { |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
40 Dir string `json:"dir"` |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
41 Originator string `json:"originator"` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 // GMAPJobKind is the unique name of an approved gauge measurements import job. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 const AGMJobKind JobKind = "agm" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 type agmJobCreator struct{} |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 func init() { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 RegisterJobCreator(AGMJobKind, agmJobCreator{}) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
52 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
53 func (agmJobCreator) AutoAccept() bool { return false } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
54 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 func (agmJobCreator) Description() string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 return "approved gauge measurements" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2173
diff
changeset
|
59 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
61 func (agmJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
62 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
63 {"gauge_measurements"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
64 {"gauges"}, |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
68 const ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
69 agmStageDoneDeleteSQL = ` |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
70 DELETE FROM waterway.gauge_measurements WHERE id IN ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
71 SELECT key |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
72 FROM import.track_imports |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
73 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
74 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
75 deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
76 )` |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
77 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 agmStageDoneSQL = ` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 UPDATE waterway.gauge_measurements SET staging_done = true |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
80 WHERE id IN ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
81 SELECT key |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
82 FROM import.track_imports |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
84 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
85 NOT deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
86 )` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 func (agmJobCreator) StageDone( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 tx *sql.Tx, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 id int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 ) error { |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
94 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
95 if err == nil { |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
96 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
97 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 return err |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 // CleanUp removes the folder containing the CSV file with the |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 // the approved gauge measurements. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 func (agm *ApprovedGaugeMeasurements) CleanUp() error { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 return os.RemoveAll(agm.Dir) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 |
4075
cb74aa69954e
Moved TimeParser to common package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4074
diff
changeset
|
107 var guessDate = common.TimeParser([]string{ |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
108 "02.01.2006 15:04", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
109 "2006-01-02T15:04:05-07:00", |
4074
eb2f949ddfa2
Rename TimeGuesser to TimeParser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4060
diff
changeset
|
110 }).Parse |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
111 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
112 type timetz struct{ time.Time } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
113 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
114 func (ttz *timetz) MarshalJSON() ([]byte, error) { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
115 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00")) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
116 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
117 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
118 type agmLine struct { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
119 id int64 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
120 Location models.Isrs `json:"fk-gauge-id"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
121 CountryCode string `json:"country-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
122 Sender string `json:"sender"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
123 LanguageCode string `json:"language-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
124 DateIssue timetz `json:"date-issue"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
125 ReferenceCode string `json:"reference-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
126 MeasureDate timetz `json:"measure-date"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
127 WaterLevel float64 `json:"water-level"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
128 DateInfo timetz `json:"date-info"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
129 SourceOrganization string `json:"source-organization"` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
130 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
131 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
132 func (a *agmLine) hasDiff(b *agmLine) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
133 const eps = 0.00001 |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
134 return a.CountryCode != b.CountryCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
135 a.Sender != b.Sender || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
136 a.LanguageCode != b.LanguageCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
137 a.ReferenceCode != b.ReferenceCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
138 math.Abs(a.WaterLevel-b.WaterLevel) > eps || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
139 a.SourceOrganization != b.SourceOrganization |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
140 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
141 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
142 type agmSummaryEntry struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
143 FKGaugeID models.Isrs `json:"fk-gauge-id"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
144 MeasureDate timetz `json:"measure-date"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
145 Versions []*agmLine `json:"versions"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
146 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
147 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
148 const ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
149 agmSelectSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
150 SELECT |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
151 id, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
152 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
153 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
154 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
155 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
156 reference_code, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
157 measure_date, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
158 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
159 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
160 source_organization |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
161 FROM waterway.gauge_measurements |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
162 WHERE |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
163 location |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
164 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
165 AND measure_date BETWEEN $6 AND $7 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
166 AND staging_done |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
167 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
168 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
169 agmInsertSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
170 INSERT INTO waterway.gauge_measurements ( |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
171 location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
172 measure_date, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
173 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
174 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
175 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
176 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
177 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
178 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
179 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
180 source_organization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
181 staging_done |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
182 ) VALUES ( |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
183 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
184 $6, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
185 $7, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
186 $8, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
187 $9, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
188 $10, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
189 $11, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
190 $12, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
191 $13, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
192 $14, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
193 false |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
194 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
195 RETURNING id` |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
196 |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
197 agmGaugeCheckSQL = ` |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
198 SELECT EXISTS( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
199 SELECT 1 FROM waterway.gauges |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
200 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
201 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
202 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
203 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
204 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
205 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
206 headerFields := []struct { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
207 idx *int |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
208 name string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
209 }{ |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
210 {fkGaugeIDIdx, "fk_gauge_id"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
211 {measureDateIdx, "measure_date"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
212 {valueIdx, "value"}, // "water_level", |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
213 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
214 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
215 nextHeader: |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
216 for i, f := range headers { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
217 h := strings.Replace(strings.ToLower( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
218 strings.TrimSpace(f)), " ", "_", -1) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
219 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
220 for j := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
221 if headerFields[j].name == h { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
222 if *headerFields[j].idx != -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
223 return fmt.Errorf( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
224 "There is more than one column namend '%s'", h) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
225 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
226 *headerFields[j].idx = i |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
227 continue nextHeader |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
228 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
229 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
230 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
231 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
232 var missing []string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
233 for i := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
234 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
235 missing = append(missing, headerFields[i].name) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
236 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
237 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
238 if len(missing) > 0 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
239 return fmt.Errorf("Missing columns: %s", strings.Join(missing, ", ")) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
240 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
241 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
242 return nil |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
243 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
244 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
245 // Do executes the actual approved gauge measurements import. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
246 func (agm *ApprovedGaugeMeasurements) Do( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
247 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
248 importID int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
249 conn *sql.Conn, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
250 feedback Feedback, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
251 ) (interface{}, error) { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
252 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
253 start := time.Now() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
254 |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
255 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv")) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
256 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
257 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
258 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
259 defer f.Close() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
260 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
261 r := csv.NewReader(bufio.NewReader(f)) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
262 r.Comma = ';' |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
263 r.ReuseRecord = true |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
264 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
265 headers, err := r.Read() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
266 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
267 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
268 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
269 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
270 var ( |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
271 fkGaugeIDIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
272 measureDateIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
273 valueIdx = -1 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
274 ) |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
275 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
276 if err := parseAGMHeaders( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
277 headers, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
278 &fkGaugeIDIdx, &measureDateIdx, &valueIdx, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
279 ); err != nil { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
280 return nil, err |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
281 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
282 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
283 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL) |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
284 if err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
285 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
286 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
287 defer gaugeCheckStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
288 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
289 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
290 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
291 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
292 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
293 defer selectStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
294 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
295 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
296 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
297 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
298 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
299 defer insertStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
300 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
301 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
302 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
303 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
304 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
305 defer trackStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
306 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
307 entries := []*agmSummaryEntry{} |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
308 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
309 checkedGauges := map[models.Isrs]bool{} |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
310 |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
311 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100} |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
312 warn := warnLimiter.Warn |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
313 defer warnLimiter.Close() |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
314 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
315 agmLines := []*agmLine{} |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
316 ignored := 0 |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
317 mdMinMax := map[models.Isrs]*[2]time.Time{} |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
318 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
319 lines: |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
320 for line := 1; ; line++ { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
321 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
322 row, err := r.Read() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
323 switch { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
324 case err == io.EOF || len(row) == 0: |
3218
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
325 feedback.Info("Read %d entries in CSV file", line-1) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
326 if ignored > 0 { |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
327 feedback.Info("%d entries ignored", ignored) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
328 } |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
329 if ignored == line-1 { |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
330 return nil, UnchangedError("No entries imported") |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
331 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
332 break lines |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
333 case err != nil: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
334 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
335 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
336 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
337 gids := row[fkGaugeIDIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
338 gid, err := models.IsrsFromString(gids) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
339 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
340 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
341 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
342 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
343 if exists, found := checkedGauges[*gid]; found { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
344 if !exists { |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
345 // Just ignore the line since we have already warned |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
346 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
347 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
348 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
349 } else { // not found in gauge cache |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
350 if err := gaugeCheckStmt.QueryRowContext( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
351 ctx, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
352 gid.CountryCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
353 gid.LoCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
354 gid.FairwaySection, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
355 gid.Orc, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
356 gid.Hectometre, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
357 ).Scan(&exists); err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
358 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
359 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
360 checkedGauges[*gid] = exists |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
361 if !exists { |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
362 warn("Ignoring data for unknown gauge %s", gid.String()) |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
363 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
364 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
365 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
366 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
367 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
368 md, err := guessDate(row[measureDateIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
369 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
370 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
371 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
372 if v := mdMinMax[*gid]; v != nil { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
373 if md.Before(v[0]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
374 v[0] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
375 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
376 if md.After(v[1]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
377 v[1] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
378 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
379 } else { |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
380 mdMinMax[*gid] = &[2]time.Time{md, md} |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
381 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
382 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
383 newSender := agm.Originator |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
384 newCountryCode := gid.CountryCode |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
385 newLanguageCode := misc.CCtoLang[gid.CountryCode] |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
386 newDateIssue := time.Now() |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
387 newReferenceCode := "ZPG" |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
388 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
389 value, err := strconv.ParseFloat(row[valueIdx], 32) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
390 if err != nil { |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
391 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
392 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
393 newValue := value |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
394 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
395 newDateInfo := newDateIssue |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
396 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
397 newSourceOrganization := newSender |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
398 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
399 agmLines = append(agmLines, newAGMLine( |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
400 *gid, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
401 newCountryCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
402 newSender, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
403 newLanguageCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
404 newDateIssue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
405 newReferenceCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
406 md, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
407 newValue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
408 newDateInfo, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
409 newSourceOrganization, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
410 )) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
411 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
412 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
413 oldGMLines := map[models.Isrs]map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
414 for gid, minMax := range mdMinMax { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
415 oldGMLines[gid], err = getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
416 ctx, selectStmt, gid, minMax[0], minMax[1]) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
417 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
418 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
419 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
420 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
421 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
422 tx, err := conn.BeginTx(ctx, nil) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
423 if err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
424 return nil, err |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
425 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
426 defer tx.Rollback() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
427 |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
428 txInsertStmt := tx.StmtContext(ctx, insertStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
429 txTrackStmt := tx.StmtContext(ctx, trackStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
430 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
431 agmLines: |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
432 for _, line := range agmLines { |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
433 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
434 var ase *agmSummaryEntry |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
435 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
436 if old := oldGMLines[line.Location]; old != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
437 ut := line.MeasureDate.Unix() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
438 if o, ok := old[ut]; ok { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
439 if !o.hasDiff(line) { // identical |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
440 // don't delete |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
441 delete(old, ut) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
442 continue agmLines |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
443 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
444 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
445 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
446 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
447 Versions: []*agmLine{o, line}, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
448 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
449 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
450 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
451 if ase == nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
452 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
453 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
454 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
455 Versions: []*agmLine{line}, |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
456 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
457 } |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
458 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
459 var newID int64 |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
460 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
461 if err := txInsertStmt.QueryRowContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
462 ctx, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
463 line.Location.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
464 line.Location.LoCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
465 line.Location.FairwaySection, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
466 line.Location.Orc, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
467 line.Location.Hectometre, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
468 line.MeasureDate.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
469 line.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
470 line.Sender, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
471 line.LanguageCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
472 line.DateIssue.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
473 line.ReferenceCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
474 line.WaterLevel, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
475 line.DateInfo.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
476 line.SourceOrganization, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
477 ).Scan(&newID); err != nil { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
478 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
479 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
480 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
481 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
482 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
483 newID, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
484 false, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
485 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
486 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
487 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
488 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
489 entries = append(entries, ase) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
490 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
491 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
492 var removed int |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
493 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
494 // Issue deletes |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
495 for _, old := range oldGMLines { |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
496 removed += len(old) |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
497 for _, line := range old { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
498 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
499 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
500 line.id, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
501 true, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
502 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
503 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
504 } |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
505 entries = append(entries, &agmSummaryEntry{ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
506 FKGaugeID: line.Location, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
507 MeasureDate: line.MeasureDate, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
508 Versions: []*agmLine{line, nil}, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
509 }) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
510 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
511 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
512 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
513 feedback.Info("Measurements to update/insert: %d", len(entries)) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
514 feedback.Info("Measurements to delete: %d", removed) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
515 |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
516 if len(entries) == 0 && removed == 0 { |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
517 return nil, UnchangedError("No changes from AGM import") |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
518 } |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
519 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
520 if err = tx.Commit(); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
521 return nil, fmt.Errorf("Commit failed: %v", err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
522 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
523 |
4060
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
524 // Sort here to mix the deletes right beside the matching inserts/updates. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
525 // This also makes the output deterministic. |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
526 sort.Slice(entries, func(i, j int) bool { |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
527 return entries[i].FKGaugeID.Less(&entries[j].FKGaugeID) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
528 }) |
15c9d4064f0f
AGM import: Mixed deletes with insert and updates to have them sorted by ISRS locations.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4049
diff
changeset
|
529 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
530 feedback.Info("Imported %d entries with changes", len(entries)) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
531 feedback.Info("Importing approved gauge measurements took %s", |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
532 time.Since(start)) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
533 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
534 return entries, nil |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
535 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
536 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
537 func getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
538 ctx context.Context, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
539 stmt *sql.Stmt, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
540 location models.Isrs, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
541 from time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
542 to time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
543 ) (map[int64]*agmLine, error) { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
544 var ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
545 oldID int64 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
546 oldCountryCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
547 oldSender string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
548 oldLanguageCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
549 oldDateIssue time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
550 oldReferenceCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
551 oldMeasureDate time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
552 oldValue float64 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
553 oldDateInfo time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
554 oldSourceOrganization string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
555 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
556 gmLines := map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
557 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
558 gms, err := stmt.QueryContext( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
559 ctx, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
560 location.CountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
561 location.LoCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
562 location.FairwaySection, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
563 location.Orc, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
564 location.Hectometre, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
565 from, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
566 to, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
567 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
568 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
569 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
570 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
571 defer gms.Close() |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
572 for gms.Next() { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
573 if err = gms.Scan( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
574 &oldID, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
575 &oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
576 &oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
577 &oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
578 &oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
579 &oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
580 &oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
581 &oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
582 &oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
583 &oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
584 ); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
585 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
586 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
587 line := newAGMLine( |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
588 location, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
589 oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
590 oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
591 oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
592 oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
593 oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
594 oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
595 oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
596 oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
597 oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
598 ) |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
599 line.id = oldID |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
600 gmLines[oldMeasureDate.Unix()] = line |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
601 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
602 if err = gms.Err(); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
603 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
604 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
605 return gmLines, nil |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
606 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
607 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
608 func newAGMLine( |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
609 location models.Isrs, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
610 countryCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
611 sender string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
612 languageCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
613 dateIssue time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
614 referenceCode string, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
615 measureDate time.Time, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
616 waterLevel float64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
617 dateInfo time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
618 sourceOrganization string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
619 ) *agmLine { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
620 return &agmLine{ |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
621 Location: location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
622 CountryCode: countryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
623 Sender: sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
624 LanguageCode: languageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
625 DateIssue: timetz{dateIssue}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
626 ReferenceCode: referenceCode, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
627 MeasureDate: timetz{measureDate}, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
628 WaterLevel: waterLevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
629 DateInfo: timetz{dateInfo}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
630 SourceOrganization: sourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
631 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
632 } |