Mercurial > gemma
annotate pkg/imports/agm.go @ 4049:4a7c2140e44b
AGM import: Generate entries for deleted items.
To detect them: Array with two lines. The second is null.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Wed, 24 Jul 2019 17:09:30 +0200 |
parents | a18bf6bc7e3c |
children | 033a8e3fec8e 15c9d4064f0f |
rev | line source |
---|---|
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
7 // Copyright (C) 2018, 2019 by via donau |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
13 // * Sascha Wilde <wilde@intevation.de> |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
14 // * Tom Gottfried <tom.gottfried@intevation.de> |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 package imports |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 import ( |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
19 "bufio" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "context" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "database/sql" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
22 "encoding/csv" |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
23 "encoding/json" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
24 "fmt" |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
25 "io" |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
26 "math" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "os" |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
28 "path/filepath" |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
29 "sort" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
30 "strconv" |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
31 "strings" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
32 "time" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/misc" |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/models" |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 type ApprovedGaugeMeasurements struct { |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
39 Dir string `json:"dir"` |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
40 Originator string `json:"originator"` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 // GMAPJobKind is the unique name of an approved gauge measurements import job. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
44 const AGMJobKind JobKind = "agm" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 type agmJobCreator struct{} |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
48 func init() { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
49 RegisterJobCreator(AGMJobKind, agmJobCreator{}) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
51 |
1754
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
52 func (agmJobCreator) AutoAccept() bool { return false } |
807569b08513
Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1743
diff
changeset
|
53 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
54 func (agmJobCreator) Description() string { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
55 return "approved gauge measurements" |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 |
2187
7c83b5277c1c
Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2173
diff
changeset
|
58 func (agmJobCreator) Create() Job { return new(ApprovedGaugeMeasurements) } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
60 func (agmJobCreator) Depends() [2][]string { |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
61 return [2][]string{ |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
62 {"gauge_measurements"}, |
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3218
diff
changeset
|
63 {"gauges"}, |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 const ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
68 agmStageDoneDeleteSQL = ` |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
69 DELETE FROM waterway.gauge_measurements WHERE id IN ( |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
70 SELECT key |
1995
59055c8301df
Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents:
1938
diff
changeset
|
71 FROM import.track_imports |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
72 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
73 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
74 deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
75 )` |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
76 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 agmStageDoneSQL = ` |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 UPDATE waterway.gauge_measurements SET staging_done = true |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
79 WHERE id IN ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
80 SELECT key |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
81 FROM import.track_imports |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 WHERE import_id = $1 AND |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
83 relation = 'waterway.gauge_measurements'::regclass AND |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
84 NOT deletion |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
85 )` |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 ) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 func (agmJobCreator) StageDone( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 tx *sql.Tx, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 id int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 ) error { |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
93 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
94 if err == nil { |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
95 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id) |
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
96 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 return err |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
100 // CleanUp removes the folder containing the CSV file with the |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 // the approved gauge measurements. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 func (agm *ApprovedGaugeMeasurements) CleanUp() error { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 return os.RemoveAll(agm.Dir) |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 } |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
105 |
1938
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
106 var guessDate = misc.TimeGuesser([]string{ |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
107 "02.01.2006 15:04", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
108 "2006-01-02T15:04:05-07:00", |
206b1dd31112
Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1781
diff
changeset
|
109 }).Guess |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
110 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
111 type timetz struct{ time.Time } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
112 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
113 func (ttz *timetz) MarshalJSON() ([]byte, error) { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
114 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00")) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
115 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
116 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
117 type agmLine struct { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
118 id int64 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
119 Location models.Isrs `json:"fk-gauge-id"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
120 CountryCode string `json:"country-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
121 Sender string `json:"sender"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
122 LanguageCode string `json:"language-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
123 DateIssue timetz `json:"date-issue"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
124 ReferenceCode string `json:"reference-code"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
125 MeasureDate timetz `json:"measure-date"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
126 WaterLevel float64 `json:"water-level"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
127 DateInfo timetz `json:"date-info"` |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
128 SourceOrganization string `json:"source-organization"` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
129 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
130 |
2173
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
131 func (a *agmLine) hasDiff(b *agmLine) bool { |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
132 const eps = 0.00001 |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
133 return a.CountryCode != b.CountryCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
134 a.Sender != b.Sender || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
135 a.LanguageCode != b.LanguageCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
136 a.ReferenceCode != b.ReferenceCode || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
137 math.Abs(a.WaterLevel-b.WaterLevel) > eps || |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
138 a.SourceOrganization != b.SourceOrganization |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
139 } |
1b1cb6cbfbf0
Approved gauge measurement imports: Suppress diff lines when old and new are identical.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2168
diff
changeset
|
140 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
141 type agmSummaryEntry struct { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
142 FKGaugeID models.Isrs `json:"fk-gauge-id"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
143 MeasureDate timetz `json:"measure-date"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
144 Versions []*agmLine `json:"versions"` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
145 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
146 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
147 const ( |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
148 agmSelectSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
149 SELECT |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
150 id, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
151 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
152 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
153 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
154 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
155 reference_code, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
156 measure_date, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
157 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
158 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
159 source_organization |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
160 FROM waterway.gauge_measurements |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
161 WHERE |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
162 location |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
163 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
164 AND measure_date BETWEEN $6 AND $7 |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
165 AND staging_done |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
166 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
167 |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
168 agmInsertSQL = ` |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
169 INSERT INTO waterway.gauge_measurements ( |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
170 location, |
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
171 validity, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
172 measure_date, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
173 country_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
174 sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
175 language_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
176 date_issue, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
177 reference_code, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
178 water_level, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
179 date_info, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
180 source_organization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
181 staging_done |
3302
ec6163c6687d
'Historicise' gauges on import
Tom Gottfried <tom@intevation.de>
parents:
3278
diff
changeset
|
182 ) VALUES ( |
2120
817cd8b89a86
Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2105
diff
changeset
|
183 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int), |
3533
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
184 COALESCE( |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
185 (SELECT validity FROM waterway.gauges |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
186 WHERE location |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
187 = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
188 AND validity @> CAST($6 AS timestamp with time zone)), |
8e083b271fca
Improve error messages if no matching gauge version found
Tom Gottfried <tom@intevation.de>
parents:
3504
diff
changeset
|
189 tstzrange(NULL, NULL)), |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
190 $6, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
191 $7, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
192 $8, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
193 $9, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
194 $10, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
195 $11, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
196 $12, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
197 $13, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
198 $14, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
199 false |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
200 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
201 RETURNING id` |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
202 |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
203 agmGaugeCheckSQL = ` |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
204 SELECT EXISTS( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
205 SELECT 1 FROM waterway.gauges |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
206 WHERE location = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int)) |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
207 ` |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
208 ) |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
209 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
210 func parseAGMHeaders(headers []string, fkGaugeIDIdx, measureDateIdx, valueIdx *int) error { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
211 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
212 headerFields := []struct { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
213 idx *int |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
214 name string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
215 }{ |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
216 {fkGaugeIDIdx, "fk_gauge_id"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
217 {measureDateIdx, "measure_date"}, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
218 {valueIdx, "value"}, // "water_level", |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
219 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
220 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
221 nextHeader: |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
222 for i, f := range headers { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
223 h := strings.Replace(strings.ToLower( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
224 strings.TrimSpace(f)), " ", "_", -1) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
225 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
226 for j := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
227 if headerFields[j].name == h { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
228 if *headerFields[j].idx != -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
229 return fmt.Errorf( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
230 "There is more than one column namend '%s'", h) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
231 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
232 *headerFields[j].idx = i |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
233 continue nextHeader |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
234 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
235 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
236 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
237 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
238 var missing []string |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
239 for i := range headerFields { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
240 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
241 missing = append(missing, headerFields[i].name) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
242 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
243 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
244 if len(missing) > 0 { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
245 return fmt.Errorf("Missing columns: %s", strings.Join(missing, ", ")) |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
246 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
247 |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
248 return nil |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
249 } |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
250 |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
251 // Do executes the actual approved gauge measurements import. |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
252 func (agm *ApprovedGaugeMeasurements) Do( |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
253 ctx context.Context, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
254 importID int64, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
255 conn *sql.Conn, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
256 feedback Feedback, |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
257 ) (interface{}, error) { |
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
258 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
259 start := time.Now() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
260 |
1743
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
261 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv")) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
262 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
263 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
264 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
265 defer f.Close() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
266 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
267 r := csv.NewReader(bufio.NewReader(f)) |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
268 r.Comma = ';' |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
269 r.ReuseRecord = true |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
270 |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
271 headers, err := r.Read() |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
272 if err != nil { |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
273 return nil, err |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
274 } |
85d0f017fbee
Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1741
diff
changeset
|
275 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
276 var ( |
2562
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
277 fkGaugeIDIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
278 measureDateIdx = -1 |
ce39e9954e85
Make upload of AGM require only "fk_gauge_id" "measure_date" and "value"
Sascha Wilde <wilde@intevation.de>
parents:
2533
diff
changeset
|
279 valueIdx = -1 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
280 ) |
1775
fcb0106ec510
Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1772
diff
changeset
|
281 |
4030
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
282 if err := parseAGMHeaders( |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
283 headers, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
284 &fkGaugeIDIdx, &measureDateIdx, &valueIdx, |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
285 ); err != nil { |
040a5dc95eb9
AGM import: Moved parsing of CVS header line to separate function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4027
diff
changeset
|
286 return nil, err |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
287 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
288 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
289 gaugeCheckStmt, err := conn.PrepareContext(ctx, agmGaugeCheckSQL) |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
290 if err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
291 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
292 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
293 defer gaugeCheckStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
294 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
295 selectStmt, err := conn.PrepareContext(ctx, agmSelectSQL) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
296 if err != nil { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
297 return nil, err |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
298 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
299 defer selectStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
300 |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
301 insertStmt, err := conn.PrepareContext(ctx, agmInsertSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
302 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
303 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
304 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
305 defer insertStmt.Close() |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
306 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
307 trackStmt, err := conn.PrepareContext(ctx, trackImportDeletionSQL) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
308 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
309 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
310 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
311 defer trackStmt.Close() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
312 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
313 entries := []*agmSummaryEntry{} |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
314 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
315 checkedGauges := map[models.Isrs]bool{} |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
316 |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
317 warnLimiter := misc.WarningLimiter{Log: feedback.Warn, MaxWarnings: 100} |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
318 warn := warnLimiter.Warn |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
319 defer warnLimiter.Close() |
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
320 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
321 agmLines := []*agmLine{} |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
322 ignored := 0 |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
323 mdMinMax := map[models.Isrs]*[2]time.Time{} |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
324 |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
325 lines: |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
326 for line := 1; ; line++ { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
327 |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
328 row, err := r.Read() |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
329 switch { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
330 case err == io.EOF || len(row) == 0: |
3218
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
331 feedback.Info("Read %d entries in CSV file", line-1) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
332 if ignored > 0 { |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
333 feedback.Info("%d entries ignored", ignored) |
c2b65a549c6f
Add minimal feedback in approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3193
diff
changeset
|
334 } |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
335 if ignored == line-1 { |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
336 return nil, UnchangedError("No entries imported") |
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
337 } |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
338 break lines |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
339 case err != nil: |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
340 return nil, fmt.Errorf("CSV parsing failed: %v", err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
341 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
342 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
343 gids := row[fkGaugeIDIdx] |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
344 gid, err := models.IsrsFromString(gids) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
345 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
346 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
347 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
348 |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
349 if exists, found := checkedGauges[*gid]; found { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
350 if !exists { |
3186
928cdc06ff37
Warn only once if gauge is unknown
Tom Gottfried <tom@intevation.de>
parents:
2853
diff
changeset
|
351 // Just ignore the line since we have already warned |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
352 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
353 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
354 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
355 } else { // not found in gauge cache |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
356 if err := gaugeCheckStmt.QueryRowContext( |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
357 ctx, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
358 gid.CountryCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
359 gid.LoCode, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
360 gid.FairwaySection, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
361 gid.Orc, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
362 gid.Hectometre, |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
363 ).Scan(&exists); err != nil { |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
364 return nil, err |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
365 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
366 checkedGauges[*gid] = exists |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
367 if !exists { |
3504
6e748f31777a
AGM/SR import. Limit logging of warnings to 100.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3502
diff
changeset
|
368 warn("Ignoring data for unknown gauge %s", gid.String()) |
3193
8329c6d3cf2a
Do not offer an import for review that did not import anything
Tom Gottfried <tom@intevation.de>
parents:
3186
diff
changeset
|
369 ignored++ |
2533
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
370 continue lines |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
371 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
372 } |
de4dc3d16647
AGM import: Check if gauge exists for a meassurement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2187
diff
changeset
|
373 |
2098
8a986d80e1c6
Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1995
diff
changeset
|
374 md, err := guessDate(row[measureDateIdx]) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
375 if err != nil { |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
376 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
377 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
378 if v := mdMinMax[*gid]; v != nil { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
379 if md.Before(v[0]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
380 v[0] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
381 } |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
382 if md.After(v[1]) { |
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
383 v[1] = md |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
384 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
385 } else { |
4035
f2d5bf42ed38
AGM import: Simplified min/max measuredate handling a little bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4032
diff
changeset
|
386 mdMinMax[*gid] = &[2]time.Time{md, md} |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
387 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
388 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
389 newSender := agm.Originator |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
390 newCountryCode := gid.CountryCode |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
391 newLanguageCode := misc.CCtoLang[gid.CountryCode] |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
392 newDateIssue := time.Now() |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
393 newReferenceCode := "ZPG" |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
394 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
395 value, err := strconv.ParseFloat(row[valueIdx], 32) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
396 if err != nil { |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
397 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
398 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
399 newValue := value |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
400 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
401 newDateInfo := newDateIssue |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
402 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
403 newSourceOrganization := newSender |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
404 |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
405 agmLines = append(agmLines, newAGMLine( |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
406 *gid, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
407 newCountryCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
408 newSender, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
409 newLanguageCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
410 newDateIssue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
411 newReferenceCode, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
412 md, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
413 newValue, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
414 newDateInfo, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
415 newSourceOrganization, |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
416 )) |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
417 } |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
418 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
419 oldGMLines := map[models.Isrs]map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
420 for gid, minMax := range mdMinMax { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
421 oldGMLines[gid], err = getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
422 ctx, selectStmt, gid, minMax[0], minMax[1]) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
423 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
424 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
425 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
426 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
427 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
428 tx, err := conn.BeginTx(ctx, nil) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
429 if err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
430 return nil, err |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
431 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
432 defer tx.Rollback() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
433 |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
434 txInsertStmt := tx.StmtContext(ctx, insertStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
435 txTrackStmt := tx.StmtContext(ctx, trackStmt) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
436 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
437 agmLines: |
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
438 for _, line := range agmLines { |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
439 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
440 var ase *agmSummaryEntry |
1780
48791416bea5
(Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1779
diff
changeset
|
441 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
442 if old := oldGMLines[line.Location]; old != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
443 ut := line.MeasureDate.Unix() |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
444 if o, ok := old[ut]; ok { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
445 if !o.hasDiff(line) { // identical |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
446 // don't delete |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
447 delete(old, ut) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
448 continue agmLines |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
449 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
450 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
451 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
452 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
453 Versions: []*agmLine{o, line}, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
454 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
455 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
456 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
457 if ase == nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
458 ase = &agmSummaryEntry{ |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
459 FKGaugeID: line.Location, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
460 MeasureDate: line.MeasureDate, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
461 Versions: []*agmLine{line}, |
3220
56b297592c0a
Handle failing INSERTs gracefully during approved gauge measurements import
Tom Gottfried <tom@intevation.de>
parents:
3219
diff
changeset
|
462 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
463 } |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
464 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
465 var newID int64 |
3502
45483dd0d801
AGM import: Don't pile up defer calls for every line in CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3389
diff
changeset
|
466 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
467 if err := txInsertStmt.QueryRowContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
468 ctx, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
469 line.Location.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
470 line.Location.LoCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
471 line.Location.FairwaySection, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
472 line.Location.Orc, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
473 line.Location.Hectometre, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
474 line.MeasureDate.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
475 line.CountryCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
476 line.Sender, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
477 line.LanguageCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
478 line.DateIssue.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
479 line.ReferenceCode, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
480 line.WaterLevel, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
481 line.DateInfo.Time, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
482 line.SourceOrganization, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
483 ).Scan(&newID); err != nil { |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
484 return nil, err |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
485 } |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
486 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
487 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
488 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
489 newID, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
490 false, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
491 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
492 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
493 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
494 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
495 entries = append(entries, ase) |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
496 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
497 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
498 var removed int |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
499 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
500 // Issue deletes |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
501 for _, key := range sortByLocationCode(oldGMLines) { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
502 old := oldGMLines[key] |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
503 removed += len(old) |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
504 for _, line := range sortByMeasureDate(old) { |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
505 if _, err := txTrackStmt.ExecContext( |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
506 ctx, importID, "waterway.gauge_measurements", |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
507 line.id, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
508 true, |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
509 ); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
510 return nil, err |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
511 } |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
512 entries = append(entries, &agmSummaryEntry{ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
513 FKGaugeID: line.Location, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
514 MeasureDate: line.MeasureDate, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
515 Versions: []*agmLine{line, nil}, |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
516 }) |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
517 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
518 } |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
519 |
4037
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
520 feedback.Info("Measurements to update/insert: %d", len(entries)) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
521 feedback.Info("Measurements to delete: %d", removed) |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
522 |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
523 if len(entries) == 0 && removed == 0 { |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
524 return nil, UnchangedError("No changes from AGM import") |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
525 } |
a18bf6bc7e3c
Detect unchanged case in AGM import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4036
diff
changeset
|
526 |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
527 if err = tx.Commit(); err != nil { |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
528 return nil, fmt.Errorf("Commit failed: %v", err) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
529 } |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
530 |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
531 feedback.Info("Imported %d entries with changes", len(entries)) |
1778
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
532 feedback.Info("Importing approved gauge measurements took %s", |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
533 time.Since(start)) |
164b46ebd60d
Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1775
diff
changeset
|
534 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
535 return entries, nil |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
536 } |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
537 |
4049
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
538 func sortByLocationCode(data map[models.Isrs]map[int64]*agmLine) []models.Isrs { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
539 keys := make([]models.Isrs, len(data)) |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
540 var i int |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
541 for key := range data { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
542 keys[i] = key |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
543 i++ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
544 } |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
545 sort.Slice(keys, func(i, j int) bool { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
546 return keys[i].Less(&keys[j]) |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
547 }) |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
548 return keys |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
549 } |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
550 |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
551 func sortByMeasureDate(data map[int64]*agmLine) []*agmLine { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
552 lines := make([]*agmLine, len(data)) |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
553 var i int |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
554 for _, line := range data { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
555 lines[i] = line |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
556 i++ |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
557 } |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
558 sort.Slice(lines, func(i, j int) bool { |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
559 return lines[i].MeasureDate.Before(lines[j].MeasureDate.Time) |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
560 }) |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
561 return lines |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
562 } |
4a7c2140e44b
AGM import: Generate entries for deleted items.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4037
diff
changeset
|
563 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
564 func getOldGMLines( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
565 ctx context.Context, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
566 stmt *sql.Stmt, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
567 location models.Isrs, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
568 from time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
569 to time.Time, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
570 ) (map[int64]*agmLine, error) { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
571 var ( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
572 oldID int64 |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
573 oldCountryCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
574 oldSender string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
575 oldLanguageCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
576 oldDateIssue time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
577 oldReferenceCode string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
578 oldMeasureDate time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
579 oldValue float64 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
580 oldDateInfo time.Time |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
581 oldSourceOrganization string |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
582 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
583 gmLines := map[int64]*agmLine{} |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
584 |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
585 gms, err := stmt.QueryContext( |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
586 ctx, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
587 location.CountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
588 location.LoCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
589 location.FairwaySection, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
590 location.Orc, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
591 location.Hectometre, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
592 from, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
593 to, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
594 ) |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
595 if err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
596 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
597 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
598 defer gms.Close() |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
599 for gms.Next() { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
600 if err = gms.Scan( |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
601 &oldID, |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
602 &oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
603 &oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
604 &oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
605 &oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
606 &oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
607 &oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
608 &oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
609 &oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
610 &oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
611 ); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
612 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
613 } |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
614 line := newAGMLine( |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
615 location, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
616 oldCountryCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
617 oldSender, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
618 oldLanguageCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
619 oldDateIssue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
620 oldReferenceCode, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
621 oldMeasureDate, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
622 oldValue, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
623 oldDateInfo, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
624 oldSourceOrganization, |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
625 ) |
4036
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
626 line.id = oldID |
e45442db19b1
First stab to make AGM imports faster by avoiding unnecessary inserts. Also delete un-updated measures in time ranges.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4035
diff
changeset
|
627 gmLines[oldMeasureDate.Unix()] = line |
4027
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
628 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
629 if err = gms.Err(); err != nil { |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
630 return nil, err |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
631 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
632 return gmLines, nil |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
633 } |
b17453420eff
Avoid doing one SELECT per line in import file
Tom Gottfried <tom@intevation.de>
parents:
4026
diff
changeset
|
634 |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
635 func newAGMLine( |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
636 location models.Isrs, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
637 countryCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
638 sender string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
639 languageCode string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
640 dateIssue time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
641 referenceCode string, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
642 measureDate time.Time, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
643 waterLevel float64, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
644 dateInfo time.Time, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
645 sourceOrganization string, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
646 ) *agmLine { |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
647 return &agmLine{ |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
648 Location: location, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
649 CountryCode: countryCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
650 Sender: sender, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
651 LanguageCode: languageCode, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
652 DateIssue: timetz{dateIssue}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
653 ReferenceCode: referenceCode, |
4023
baa51bb82364
AGM import: parse CSV to completion before requesting database
Tom Gottfried <tom@intevation.de>
parents:
3533
diff
changeset
|
654 MeasureDate: timetz{measureDate}, |
2105
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
655 WaterLevel: waterLevel, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
656 DateInfo: timetz{dateInfo}, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
657 SourceOrganization: sourceOrganization, |
58a28715e386
Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2098
diff
changeset
|
658 } |
1741
44398a8bdf94
Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
659 } |