Mercurial > gemma
annotate pkg/controllers/srimports.go @ 1226:2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
author | Sascha L. Teichmann <sascha.teichmann@intevation.de> |
---|---|
date | Mon, 19 Nov 2018 17:56:44 +0100 |
parents | 4d7c44f7044e |
children | 737e1acea1f1 |
rev | line source |
---|---|
1017
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
1 // This is Free Software under GNU Affero General Public License v >= 3.0 |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
2 // without warranty, see README.md and license for details. |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
3 // |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
4 // SPDX-License-Identifier: AGPL-3.0-or-later |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
5 // License-Filename: LICENSES/AGPL-3.0.txt |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
6 // |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
7 // Copyright (C) 2018 by via donau |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
8 // – Österreichische Wasserstraßen-Gesellschaft mbH |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
9 // Software engineering by Intevation GmbH |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
10 // |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
11 // Author(s): |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de> |
a244b18cb916
Added GNU Affero General Public License.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1010
diff
changeset
|
13 |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
14 package controllers |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
15 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
16 import ( |
1225
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
17 "archive/zip" |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
18 "bufio" |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
19 "database/sql" |
1221
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
20 "encoding/hex" |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
21 "fmt" |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
22 "io" |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
23 "io/ioutil" |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
24 "log" |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
25 "net/http" |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
26 "os" |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
27 "path/filepath" |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
28 "sync" |
1221
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
29 "time" |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
30 |
979
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
31 "gemma.intevation.de/gemma/pkg/auth" |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
32 "gemma.intevation.de/gemma/pkg/common" |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
33 "gemma.intevation.de/gemma/pkg/config" |
979
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
34 "gemma.intevation.de/gemma/pkg/imports" |
1221
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
35 "gemma.intevation.de/gemma/pkg/misc" |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
36 "gemma.intevation.de/gemma/pkg/models" |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
37 ) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
38 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
39 const ( |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
40 maxSoundingResultSize = 25 * 1024 * 1024 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
41 soundingResultName = "soundingresult" |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
42 ) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
43 |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
44 func fetchSoundingResult(req *http.Request) (string, error) { |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
45 |
1221
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
46 // Check first if we have a token. |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
47 if token := req.FormValue("token"); token != "" { |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
48 if _, err := hex.DecodeString(token); err != nil { |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
49 return "", err |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
50 } |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
51 dir := config.TmpDir() |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
52 if dir == "" { |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
53 dir = os.TempDir() |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
54 } |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
55 // XXX: This should hopefully be race-free enough. |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
56 now := time.Now().Format("2006-15-04-05") |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
57 dst := filepath.Join(dir, soundingResultName+"-"+token+"-"+now) |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
58 if err := misc.UnmakeTempFile(token, dst); err != nil { |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
59 return "", err |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
60 } |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
61 } |
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
62 |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
63 return storeSoundingResult(req) |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
64 } |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
65 |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
66 func storeSoundingResult(req *http.Request) (string, error) { |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
67 |
1221
c193649d4f11
Add an area for temp uploads on the server to be addressed
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1203
diff
changeset
|
68 // Check for direct upload. |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
69 f, _, err := req.FormFile(soundingResultName) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
70 if err != nil { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
71 return "", err |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
72 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
73 defer f.Close() |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
74 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
75 dir, err := ioutil.TempDir(config.TmpDir(), soundingResultName) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
76 if err != nil { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
77 return "", err |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
78 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
79 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
80 o, err := os.Create(filepath.Join(dir, "sr.zip")) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
81 if err != nil { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
82 os.RemoveAll(dir) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
83 return "", err |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
84 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
85 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
86 out := bufio.NewWriter(o) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
87 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
88 if _, err = io.Copy(out, io.LimitReader(f, maxSoundingResultSize)); err != nil { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
89 o.Close() |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
90 os.RemoveAll(dir) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
91 return "", err |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
92 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
93 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
94 if err = out.Flush(); err != nil { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
95 o.Close() |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
96 os.RemoveAll(dir) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
97 return "", err |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
98 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
99 |
984
3c9ea8ab856a
Fixed small glitches in sounding result importer.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
979
diff
changeset
|
100 return dir, nil |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
101 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
102 |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
103 func importSoundingResult(rw http.ResponseWriter, req *http.Request) { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
104 |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
105 dir, err := fetchSoundingResult(req) |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
106 if err != nil { |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
107 log.Printf("error: %v\n", err) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
108 http.Error(rw, "error: "+err.Error(), http.StatusInternalServerError) |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
109 return |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
110 } |
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
111 |
979
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
112 session, _ := auth.GetSession(req) |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
113 |
998
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
114 jobID, err := imports.AddJob(imports.SRJobKind, session.User, dir) |
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
115 if err != nil { |
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
116 log.Printf("error: %v\n", err) |
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
117 http.Error(rw, "error: "+err.Error(), http.StatusInternalServerError) |
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
118 return |
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
119 } |
75e65599ea52
Persist job queue in database. WIP.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
991
diff
changeset
|
120 |
1010
8f23ec811afb
Fixed and harmonized wording in importer queue a bit.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
998
diff
changeset
|
121 log.Printf("info: added import #%d to queue\n", jobID) |
979
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
122 |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
123 result := struct { |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
124 ID int64 `json:"id"` |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
125 }{ |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
126 ID: jobID, |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
127 } |
7934b5c1a910
Finally enqueue sounding result import job to import jobs.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
978
diff
changeset
|
128 SendJSON(rw, http.StatusCreated, &result) |
978
544a5cfe07cd
Started with endpoint for uploading sounding result and trigger respective import job.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff
changeset
|
129 } |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
130 |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
131 func loadMeta(f *zip.File) (*models.SoundingResultMeta, error) { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
132 r, err := f.Open() |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
133 if err != nil { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
134 return nil, err |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
135 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
136 defer r.Close() |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
137 var m models.SoundingResultMeta |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
138 return &m, m.Decode(r) |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
139 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
140 |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
141 func uploadSoundingResult( |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
142 _ interface{}, |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
143 req *http.Request, |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
144 conn *sql.Conn, |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
145 ) (jr JSONResult, err error) { |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
146 |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
147 var dir string |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
148 if dir, err = storeSoundingResult(req); err != nil { |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
149 return |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
150 } |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
151 |
1225
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
152 srFile := filepath.Join(dir, "sr.zip") |
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
153 |
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
154 var zr *zip.ReadCloser |
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
155 if zr, err = zip.OpenReader(srFile); err != nil { |
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
156 return |
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
157 } |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
158 var once sync.Once |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
159 closeOnce := func() { zr.Close() } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
160 defer once.Do(closeOnce) |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
161 |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
162 var messages []string |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
163 |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
164 var result struct { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
165 Token string `json:"token"` |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
166 Meta interface{} `json:"meta,omitempty"` |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
167 Messages []string `json:"messages,omitempty"` |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
168 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
169 |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
170 if common.FindInZIP(zr, ".xyz") == nil { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
171 messages = append(messages, "no .xyz file found.") |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
172 } |
1225
4d7c44f7044e
Factored out som zip lookup code to be reusable in sounding result upload controller.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1224
diff
changeset
|
173 |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
174 if mj := common.FindInZIP(zr, "meta.json"); mj == nil { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
175 messages = append(messages, "no 'meta.json' file found.") |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
176 } else { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
177 if meta, err := loadMeta(mj); err != nil { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
178 messages = append(messages, |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
179 fmt.Sprintf("'meta.json' found but invalid: %v", err)) |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
180 } else { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
181 errs := meta.Validate(conn, req.Context()) |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
182 for _, err := range errs { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
183 messages = append(messages, |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
184 fmt.Sprintf("invalid 'meta.json': %v", err)) |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
185 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
186 result.Meta = meta |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
187 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
188 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
189 once.Do(closeOnce) |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
190 |
1226
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
191 if result.Token, err = misc.MakeTempFile(dir); err != nil { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
192 if err2 := os.RemoveAll(dir); err2 != nil { |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
193 log.Printf("error: %v\n", err2) |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
194 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
195 return |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
196 } |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
197 |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
198 result.Messages = messages |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
199 |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
200 jr = JSONResult{ |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
201 Code: http.StatusCreated, |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
202 Result: &result, |
2e65e8ddacab
Finished the sounding result upload controller to temp uploads.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1225
diff
changeset
|
203 } |
1224
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
204 return |
bc4b642c8d04
Started with an upload sounding result to temp upload area.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
1221
diff
changeset
|
205 } |