Mercurial > gemma

annotate pkg/imports/agm.go @ 2168:b868cb653c4d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Import queue: The job kind parameter when creating a new job from a string was never used.
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Fri, 08 Feb 2019 15:39:17 +0100
parents 817cd8b89a86
children 1b1cb6cbfbf0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
1 // This is Free Software under GNU Affero General Public License v >= 3.0
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
2 // without warranty, see README.md and license for details.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
3 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
4 // SPDX-License-Identifier: AGPL-3.0-or-later
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
5 // License-Filename: LICENSES/AGPL-3.0.txt
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
6 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
7 // Copyright (C) 2018 by via donau
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
8 // – Österreichische Wasserstraßen-Gesellschaft mbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
9 // Software engineering by Intevation GmbH
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10 //
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
11 // Author(s):
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
13
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14 package imports
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import (
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
17 "bufio"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 "context"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19 "database/sql"
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
20 "encoding/csv"
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
21 "encoding/json"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
22 "fmt"
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
23 "io"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 "os"
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
25 "path/filepath"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
26 "strconv"
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
27 "strings"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
28 "time"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30 "gemma.intevation.de/gemma/pkg/common"
1938
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
31 "gemma.intevation.de/gemma/pkg/misc"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
32 "gemma.intevation.de/gemma/pkg/models"
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
33 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
34
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
35 type ApprovedGaugeMeasurements struct {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
36 Dir string `json:"dir"`
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
37 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
38
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39 // GMAPJobKind is the unique name of an approved gauge measurements import job.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40 const AGMJobKind JobKind = "agm"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
41
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
42 type agmJobCreator struct{}
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
44 func init() {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
45 RegisterJobCreator(AGMJobKind, agmJobCreator{})
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
46 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
47
1754
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1743
diff changeset
48 func (agmJobCreator) AutoAccept() bool { return false }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1743
diff changeset
49
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50 func (agmJobCreator) Description() string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
51 return "approved gauge measurements"
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
52 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
53
2168
b868cb653c4d Import queue: The job kind parameter when creating a new job from a string was never used.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2120
diff changeset
54 func (agmJobCreator) Create(data string) (Job, error) {
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
55 agm := new(ApprovedGaugeMeasurements)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
56 if err := common.FromJSONString(data, agm); err != nil {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
57 return nil, err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
58 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
59 return agm, nil
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
61
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
62 func (agmJobCreator) Depends() []string {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
63 return []string{
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
64 "gauges",
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
65 "gauge_measurements",
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
66 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
67 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
68
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
69 const (
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
70 // delete the old and keep the new measures.
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
71 agmStageDoneDeleteSQL = `
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
72 WITH staged AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
73 SELECT key
1995
59055c8301df Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents: 1938
diff changeset
74 FROM import.track_imports
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
75 WHERE import_id = $1 AND
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
76 relation = 'waterway.gauge_measurements'::regclass
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
77 ),
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
78 to_delete AS (
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
79 SELECT o.id AS id
1781
0a53c33bc7b2 Fixed broken SQL statement (once again).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1780
diff changeset
80 FROM waterway.gauge_measurements o
0a53c33bc7b2 Fixed broken SQL statement (once again).
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1780
diff changeset
81 JOIN waterway.gauge_measurements n
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
82 ON n.fk_gauge_id = o.fk_gauge_id AND n.measure_date = o.measure_date
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
83 WHERE n.id IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
84 AND o.id NOT IN (SELECT key FROM staged)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
85 )
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
86 DELETE FROM waterway.gauge_measurements WHERE id IN (SELECT id from to_delete)`
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
87
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
88 agmStageDoneSQL = `
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
89 UPDATE waterway.gauge_measurements SET staging_done = true
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
90 WHERE id IN (
1995
59055c8301df Move import queue to its own database namespace
Tom Gottfried <tom@intevation.de>
parents: 1938
diff changeset
91 SELECT key FROM import.track_imports
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
92 WHERE import_id = $1 AND
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
93 relation = 'waterway.gauge_measurements'::regclass)`
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94 )
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
96 func (agmJobCreator) StageDone(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
97 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
98 tx *sql.Tx,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
99 id int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
100 ) error {
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
101 _, err := tx.ExecContext(ctx, agmStageDoneDeleteSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
102 if err == nil {
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
103 _, err = tx.ExecContext(ctx, agmStageDoneSQL, id)
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
104 }
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
105 return err
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
106 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
107
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
108 // CleanUp removes the folder containing the CSV file with the
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
109 // the approved gauge measurements.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
110 func (agm *ApprovedGaugeMeasurements) CleanUp() error {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
111 return os.RemoveAll(agm.Dir)
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
112 }
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
113
1938
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
114 var guessDate = misc.TimeGuesser([]string{
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
115 "02.01.2006 15:04",
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
116 "2006-01-02T15:04:05-07:00",
206b1dd31112 Moved time guessing to misc code to be recycled.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1781
diff changeset
117 }).Guess
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
118
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
119 type timetz struct{ time.Time }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
120
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
121 func (ttz *timetz) MarshalJSON() ([]byte, error) {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
122 return json.Marshal(ttz.Time.Format("2006-01-02T15:04:05-07:00"))
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
123 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
124
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
125 type agmLine struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
126 CountryCode string `json:"country-code"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
127 Sender string `json:"sender"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
128 LanguageCode string `json:"language-code"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
129 DateIssue timetz `json:"date-issue"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
130 ReferenceCode string `json:"reference-code"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
131 WaterLevel float64 `json:"water-level"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
132 Predicted bool `json:"predicted"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
133 ValueMin *float64 `json:"value-min"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
134 ValueMax *float64 `json:"value-max"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
135 DateInfo timetz `json:"date-info"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
136 SourceOrganization string `json:"source-organization"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
137 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
138
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
139 type agmSummaryEntry struct {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
140 FKGaugeID models.Isrs `json:"fk-gauge-id"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
141 MeasureDate timetz `json:"measure-date"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
142 Versions []*agmLine `json:"versions"`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
143 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
144
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
145 const (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
146 agmSelectSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
147 SELECT
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
148 id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
149 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
150 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
151 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
152 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
153 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
154 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
155 predicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
156 value_min,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
157 value_max,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
158 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
159 source_organization
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
160 FROM waterway.gauge_measurements
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
161 WHERE
2120
817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2105
diff changeset
162 fk_gauge_id = ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int) AND
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
163 measure_date = $6 AND staging_done`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
164
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
165 agmInsertSQL = `
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
166 INSERT INTO waterway.gauge_measurements (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
167 fk_gauge_id,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
168 measure_date,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
169 country_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
170 sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
171 language_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
172 date_issue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
173 reference_code,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
174 water_level,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
175 predicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
176 value_min,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
177 value_max,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
178 date_info,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
179 source_organization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
180 is_waterlevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
181 staging_done
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
182 ) VALUES(
2120
817cd8b89a86 Approved gauge measuments import: Give type hints when insertinf/querying isrs location codes.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2105
diff changeset
183 ($1::char(2), $2::char(3), $3::char(5), $4::char(5), $5::int),
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
184 $6,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
185 $7,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
186 $8,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
187 $9,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
188 $10,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
189 $11,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
190 $12,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
191 $13,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
192 $14,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
193 $15,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
194 $16,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
195 $17,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
196 true,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
197 false
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
198 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
199 RETURNING id`
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
200 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
201
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
202 // Do executes the actual approved gauge measurements import.
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
203 func (agm *ApprovedGaugeMeasurements) Do(
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
204 ctx context.Context,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
205 importID int64,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
206 conn *sql.Conn,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
207 feedback Feedback,
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 ) (interface{}, error) {
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
210 start := time.Now()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
211
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
212 f, err := os.Open(filepath.Join(agm.Dir, "agm.csv"))
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
213 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
214 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
215 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
216 defer f.Close()
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
217
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
218 r := csv.NewReader(bufio.NewReader(f))
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
219 r.Comma = ';'
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
220 r.ReuseRecord = true
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
221
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
222 headers, err := r.Read()
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
223 if err != nil {
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
224 return nil, err
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
225 }
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
226
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
227 var (
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
228 fkGaugeIDIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
229 measureDateIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
230 fromIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
231 languageCodeIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
232 countryCodeIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
233 dateIssueIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
234 referenceCodeIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
235 valueIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
236 predictedIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
237 valueMinIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
238 valueMaxIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
239 dateInfoIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
240 originatorIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
241 unitIdx = -1
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
242 )
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
243
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
244 headerFields := []struct {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
245 idx *int
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
246 name string
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
247 }{
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
248 {&fkGaugeIDIdx, "fk_gauge_id"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
249 {&measureDateIdx, "measure_date"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
250 {&fromIdx, "from"}, // "sender",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
251 {&languageCodeIdx, "language_code"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
252 {&countryCodeIdx, "country_code"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
253 {&dateIssueIdx, "date_issue"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
254 {&referenceCodeIdx, "reference_code"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
255 {&valueIdx, "value"}, // "water_level",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
256 {&predictedIdx, "predicted"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
257 // "is_waterlevel",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
258 {&valueMinIdx, "value_min"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
259 {&valueMaxIdx, "value_max"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
260 {&dateInfoIdx, "date_info"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
261 {&originatorIdx, "originator"}, // "source_organization",
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
262 {&unitIdx, "unit"},
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
263 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
264
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
265 nextHeader:
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
266 for i, f := range headers {
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
267 h := strings.Replace(strings.ToLower(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
268 strings.TrimSpace(f)), " ", "_", -1)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
269
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
270 for j := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
271 if headerFields[j].name == h {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
272 if *headerFields[j].idx != -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
273 return nil, fmt.Errorf(
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
274 "There is more than one column namend '%s'", h)
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
275 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
276 *headerFields[j].idx = i
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
277 continue nextHeader
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
278 }
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
279 }
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
280 }
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
281
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
282 var missing []string
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
283 for i := range headerFields {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
284 if headerFields[i].name != "unit" && *headerFields[i].idx == -1 {
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
285 missing = append(missing, headerFields[i].name)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
286 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
287 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
288 if len(missing) > 0 {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
289 return nil, fmt.Errorf("Missing columns: %s", strings.Join(missing, ", "))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
290 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
291
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
292 inCm, _ := rescale("cm")
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
293 scaler := func(row []string) (func(float32) float32, error) {
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
294 if unitIdx == -1 {
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
295 return inCm, nil
1775
fcb0106ec510 Gauge measurement import: Added reference_code column.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1772
diff changeset
296 }
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
297 unit := row[unitIdx]
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
298 if unit == "cm" {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
299 return inCm, nil
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
300 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
301 s, err := rescale(unit)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
302 return s, err
1743
85d0f017fbee Approved gauges measurements: Open CSV and read headers.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1741
diff changeset
303 }
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
304
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
305 tx, err := conn.BeginTx(ctx, nil)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
306 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
307 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
308 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
309 defer tx.Rollback()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
310
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
311 selectStmt, err := tx.PrepareContext(ctx, agmSelectSQL)
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
312 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
313 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
314 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
315 defer selectStmt.Close()
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
316 insertStmt, err := tx.PrepareContext(ctx, agmInsertSQL)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
317 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
318 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
319 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
320 defer insertStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
321 trackStmt, err := tx.PrepareContext(ctx, trackImportSQL)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
322 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
323 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
324 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
325 defer trackStmt.Close()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
326
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
327 entries := []*agmSummaryEntry{}
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
328
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
329 lines:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
330 for line := 1; ; line++ {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
331
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
332 row, err := r.Read()
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
333 switch {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
334 case err == io.EOF || len(row) == 0:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
335 break lines
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
336 case err != nil:
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
337 return nil, fmt.Errorf("CSV parsing failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
338 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
339 convert, err := scaler(row)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
340 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
341 return nil, fmt.Errorf("line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
342 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
343
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
344 gids := row[fkGaugeIDIdx]
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
345 gid, err := models.IsrsFromString(gids)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
346 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
347 return nil, fmt.Errorf("Invalid ISRS code line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
348 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
349
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
350 md, err := guessDate(row[measureDateIdx])
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
351 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
352 return nil, fmt.Errorf("Invalid 'measure_date' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
353 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
354
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
355 var (
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
356 oldID int64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
357 oldCountryCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
358 oldSender string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
359 oldLanguageCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
360 oldDateIssue time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
361 oldReferenceCode string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
362 oldValue float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
363 oldPredicted bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
364 oldValueMin sql.NullFloat64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
365 oldValueMax sql.NullFloat64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
366 oldDateInfo time.Time
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
367 oldSourceOrganization string
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
368 )
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
369
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
370 err = selectStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
371 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
372 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
373 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
374 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
375 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
376 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
377 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
378 ).Scan(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
379 &oldID,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
380 &oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
381 &oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
382 &oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
383 &oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
384 &oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
385 &oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
386 &oldPredicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
387 &oldValueMin,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
388 &oldValueMax,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
389 &oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
390 &oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
391 )
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
392
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
393 var newEntry bool
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
394 switch {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
395 case err == sql.ErrNoRows:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
396 // Complete new one
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
397 newEntry = true
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
398 case err != nil:
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
399 return nil, err
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
400 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
401
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
402 newSender := row[fromIdx]
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
403 newLanguageCode := row[languageCodeIdx]
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
404 newCountryCode := row[countryCodeIdx]
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
405
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
406 dis, err := guessDate(row[dateIssueIdx])
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
407 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
408 return nil, fmt.Errorf("Invalid 'date_issue' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
409 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
410 newDateIssue := dis
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
411
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
412 newReferenceCode := row[referenceCodeIdx]
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
413
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
414 value, err := strconv.ParseFloat(row[valueIdx], 32)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
415 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
416 return nil, fmt.Errorf("Invalid 'value' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
417 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
418 newValue := float64(convert(float32(value)))
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
419
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
420 newPredicted := strings.ToLower(row[predictedIdx]) == "true"
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
421
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
422 var newValueMin sql.NullFloat64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
423 if vm := row[valueMinIdx]; vm != "" {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
424 valueMin, err := strconv.ParseFloat(vm, 32)
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
425 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
426 return nil, fmt.Errorf("Invalid 'value_min' line %d: %v", line, err)
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
427 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
428 newValueMin = sql.NullFloat64{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
429 Float64: float64(convert(float32(valueMin))),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
430 Valid: true,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
431 }
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
432 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
433
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
434 var newValueMax sql.NullFloat64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
435 if vm := row[valueMaxIdx]; vm != "" {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
436 valueMax, err := strconv.ParseFloat(vm, 32)
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
437 if err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
438 return nil, fmt.Errorf("Invalid 'value_max' line %d: %v", line, err)
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
439 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
440 newValueMax = sql.NullFloat64{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
441 Float64: float64(convert(float32(valueMax))),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
442 Valid: true,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
443 }
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
444 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
445
2098
8a986d80e1c6 Approved gauge measuremnet imports: Removed an indirection layer accessing the columns of the CSV file.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1995
diff changeset
446 din, err := guessDate(row[dateInfoIdx])
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
447 if err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
448 return nil, fmt.Errorf("Invalid 'date_info' line %d: %v", line, err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
449 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
450 newDateInfo := din
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
451
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
452 newSourceOrganization := row[originatorIdx]
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
453
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
454 var newID int64
1780
48791416bea5 (Approved) gauge measurement import: Fixed row level security.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1779
diff changeset
455
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
456 if err := insertStmt.QueryRowContext(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
457 ctx,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
458 gid.CountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
459 gid.LoCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
460 gid.FairwaySection,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
461 gid.Orc,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
462 gid.Hectometre,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
463 md,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
464 newCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
465 newSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
466 newLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
467 newDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
468 newReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
469 newValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
470 newPredicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
471 newValueMin,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
472 newValueMax,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
473 newDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
474 newSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
475 ).Scan(&newID); err != nil {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
476 return nil, err
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
477 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
478 if _, err := trackStmt.ExecContext(
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
479 ctx, importID, "waterway.gauge_measurements", newID,
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
480 ); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
481 return nil, err
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
482 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
483
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
484 ase := &agmSummaryEntry{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
485 FKGaugeID: *gid,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
486 MeasureDate: timetz{md},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
487 Versions: []*agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
488 newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
489 newCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
490 newSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
491 newLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
492 newDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
493 newReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
494 newValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
495 newPredicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
496 newValueMin,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
497 newValueMax,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
498 newDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
499 newSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
500 ),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
501 },
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
502 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
503
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
504 if !newEntry {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
505 ase.Versions = []*agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
506 newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
507 oldCountryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
508 oldSender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
509 oldLanguageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
510 oldDateIssue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
511 oldReferenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
512 oldValue,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
513 oldPredicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
514 oldValueMin,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
515 oldValueMax,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
516 oldDateInfo,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
517 oldSourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
518 ),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
519 ase.Versions[0],
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
520 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
521 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
522 entries = append(entries, ase)
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
523 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
524
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
525 if err := tx.Commit(); err != nil {
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
526 return nil, fmt.Errorf("Commit failed: %v", err)
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
527 }
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
528
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
529 feedback.Info("Importing approved gauge measurements took %s",
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
530 time.Since(start))
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
531
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
532 return entries, nil
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
533 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
534
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
535 func newAGMLine(
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
536 countryCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
537 sender string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
538 languageCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
539 dateIssue time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
540 referenceCode string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
541 waterLevel float64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
542 predicted bool,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
543 valueMin sql.NullFloat64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
544 valueMax sql.NullFloat64,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
545 dateInfo time.Time,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
546 sourceOrganization string,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
547 ) *agmLine {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
548 nilFloat := func(v sql.NullFloat64) *float64 {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
549 var p *float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
550 if v.Valid {
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
551 p = &v.Float64
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
552 }
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
553 return p
1778
164b46ebd60d Approved gauge measurement import: Implemented. TODO: Fix staging.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1775
diff changeset
554 }
2105
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
555 return &agmLine{
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
556 CountryCode: countryCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
557 Sender: sender,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
558 LanguageCode: languageCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
559 DateIssue: timetz{dateIssue},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
560 ReferenceCode: referenceCode,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
561 WaterLevel: waterLevel,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
562 Predicted: predicted,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
563 ValueMin: nilFloat(valueMin),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
564 ValueMax: nilFloat(valueMax),
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
565 DateInfo: timetz{dateInfo},
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
566 SourceOrganization: sourceOrganization,
58a28715e386 Approved gauge measurement import: Added diff-summary. XXX: May be broken!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 2098
diff changeset
567 }
1741
44398a8bdf94 Approved gauge measurements: Added a stub to upload a CSV file for parsing.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
568 }