Mercurial > gemma
annotate pkg/imports/ugm.go @ 5520:05db984d3db1
Improve performance of bottleneck area calculation
Avoid buffer calculations by replacing them with simple distance comparisons
and calculate the boundary of the result geometry only once per iteration.
In some edge cases with very large numbers of iterations, this reduced
the runtime of a bottleneck import by a factor of more than twenty.
author | Tom Gottfried <tom@intevation.de> |
---|---|
date | Thu, 21 Oct 2021 19:50:39 +0200 |
parents | 56c589f7435d |
children | 6270951dda28 |
rev | line source |
---|---|
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
2 // without warranty, see README.md and license for details. |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
3 // |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
6 // |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
9 // Software engineering by Intevation GmbH |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
10 // |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
11 // Author(s): |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
13 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package imports |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
17 "context" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "database/sql" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
19 "errors" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
20 "os" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
21 "path/filepath" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "gemma.intevation.de/gemma/pkg/soap" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "gemma.intevation.de/gemma/pkg/soap/nts" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 ) |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
27 // UploadedGaugeMeasurement is an Jon to extract gauge measurement data |
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
28 // from an uploaded XML file and stores it into the database. |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
29 type UploadedGaugeMeasurement struct { |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 Dir string `json:"dir"` |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
31 } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
32 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
33 // UGMJobKind is the unique name of this import job type. |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
34 const UGMJobKind JobKind = "ugm" |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
35 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
36 type ugmJobCreator struct{} |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 func init() { RegisterJobCreator(UGMJobKind, ugmJobCreator{}) } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
39 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
40 func (ugmJobCreator) Description() string { return "uploaded gauge measurements" } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
42 func (ugmJobCreator) Create() Job { return new(UploadedGaugeMeasurement) } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 |
3219
4acbee65275d
Import queue: Split locked dependencies in exclusively and multiple uses.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2278
diff
changeset
|
44 func (ugmJobCreator) Depends() [2][]string { return gmJobCreator{}.Depends() } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
46 func (ugmJobCreator) AutoAccept() bool { return true } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
47 |
5035
56c589f7435d
Enhance comments on implementations of StageDone()
Tom Gottfried <tom@intevation.de>
parents:
5034
diff
changeset
|
48 // StageDone is a NOP for gauge measurements imports. |
5034
59a99655f34d
Added feedback support for StageDone.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4182
diff
changeset
|
49 func (ugmJobCreator) StageDone(context.Context, *sql.Tx, int64, Feedback) error { return nil } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
50 |
4182
49012340336c
Made 'golint' finally happy with imports package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
4177
diff
changeset
|
51 // CleanUp removes the temporary files from the filesystem. |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
52 func (ugm *UploadedGaugeMeasurement) CleanUp() error { return os.RemoveAll(ugm.Dir) } |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
53 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
54 // Do executes the actual uploaded gauge measurement import. |
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
55 func (ugm *UploadedGaugeMeasurement) Do( |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
56 ctx context.Context, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
57 importID int64, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
58 conn *sql.Conn, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
59 feedback Feedback, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
60 ) (interface{}, error) { |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
61 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
62 fetch := func() ([]*nts.RIS_Message_Type, error) { |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
63 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
64 var dst nts.Get_messages_result |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
65 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
66 if err := soap.ValidateFile( |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
67 filepath.Join(ugm.Dir, "data.xml"), |
2278
a3e7e727209a
UGM import: Use special schema.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2249
diff
changeset
|
68 "NtS.xsd", |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 &dst, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 ); err != nil { |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 return nil, err |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
74 if len(dst.Result_message) == 0 { |
4177
8b75ac5e243e
Made 'staticcheck' happy with pgxutils package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
3219
diff
changeset
|
75 return nil, errors.New("no gauge measurements found") |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 } |
2249
35db2e11c966
Uploaded gauge measurement: Implemented in terms of normal gauge measurement import.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
2246
diff
changeset
|
77 return dst.Result_message, nil |
2245
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 } |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 return storeGaugeMeasurements( |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 ctx, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 importID, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 fetch, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 conn, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 feedback, |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 ) |
7d784840a9a7
Uploaded gauge measurements: Initial commit. TODO: Add endpoint.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 } |