gemma: pkg/controllers/srimports.go annotate

annotate pkg/controllers/srimports.go @ 1221:c193649d4f11

Add an area for temp uploads on the server to be addressed by tokens. If unused they will me thrown away after 45 minutes. There could be max 100 of them. If there are more to upload the oldest are removed first.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Mon, 19 Nov 2018 16:15:30 +0100
parents	49eead4fad3a
children	bc4b642c8d04

rev	line source
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	3 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	6 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	10 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	11 // Author(s):
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	13
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package controllers
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "bufio"
1221 c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	18 "encoding/hex"
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "io"
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "io/ioutil"
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "log"
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	22 "net/http"
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	23 "os"
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24 "path/filepath"
1221 c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	25 "time"
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
979 7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	27 "gemma.intevation.de/gemma/pkg/auth"
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 "gemma.intevation.de/gemma/pkg/config"
979 7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	29 "gemma.intevation.de/gemma/pkg/imports"
1221 c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	30 "gemma.intevation.de/gemma/pkg/misc"
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 )
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 const (
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 maxSoundingResultSize = 25 * 1024 * 1024
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 soundingResultName = "soundingresult"
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	36 )
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	37
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38 func downloadSoundingResult(req *http.Request) (string, error) {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39
1221 c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	40 // Check first if we have a token.
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	41 if token := req.FormValue("token"); token != "" {
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	42 if _, err := hex.DecodeString(token); err != nil {
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	43 return "", err
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	44 }
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	45 dir := config.TmpDir()
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	46 if dir == "" {
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	47 dir = os.TempDir()
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	48 }
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	49 // XXX: This should hopefully be race-free enough.
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	50 now := time.Now().Format("2006-15-04-05")
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	51 dst := filepath.Join(dir, soundingResultName+"-"+token+"-"+now)
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	52 if err := misc.UnmakeTempFile(token, dst); err != nil {
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	53 return "", err
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	54 }
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	55 }
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	56
c193649d4f11 Add an area for temp uploads on the server to be addressed Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1203 diff changeset	57 // Check for direct upload.
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	58 f, _, err := req.FormFile(soundingResultName)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	59 if err != nil {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	60 return "", err
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	62 defer f.Close()
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	63
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	64 dir, err := ioutil.TempDir(config.TmpDir(), soundingResultName)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	65 if err != nil {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	66 return "", err
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	67 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	68
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	69 o, err := os.Create(filepath.Join(dir, "sr.zip"))
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	70 if err != nil {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	71 os.RemoveAll(dir)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	72 return "", err
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	73 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	74
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	75 out := bufio.NewWriter(o)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	76
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	77 if _, err = io.Copy(out, io.LimitReader(f, maxSoundingResultSize)); err != nil {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	78 o.Close()
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	79 os.RemoveAll(dir)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	80 return "", err
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	81 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	82
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	83 if err = out.Flush(); err != nil {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	84 o.Close()
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	85 os.RemoveAll(dir)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	86 return "", err
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	87 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	88
984 3c9ea8ab856a Fixed small glitches in sounding result importer. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	89 return dir, nil
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92 func importSoundingResult(rw http.ResponseWriter, req *http.Request) {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	93
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	94 dir, err := downloadSoundingResult(req)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	95 if err != nil {
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	96 log.Printf("error: %v\n", err)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	97 http.Error(rw, "error: "+err.Error(), http.StatusInternalServerError)
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	98 return
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	99 }
544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	100
979 7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	101 session, _ := auth.GetSession(req)
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	102
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	103 jobID, err := imports.AddJob(imports.SRJobKind, session.User, dir)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	104 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	105 log.Printf("error: %v\n", err)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	106 http.Error(rw, "error: "+err.Error(), http.StatusInternalServerError)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	107 return
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	108 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	109
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	110 log.Printf("info: added import #%d to queue\n", jobID)
979 7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	111
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	112 result := struct {
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	113 ID int64 `json:"id"`
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	114 }{
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	115 ID: jobID,
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	116 }
7934b5c1a910 Finally enqueue sounding result import job to import jobs. Sascha L. Teichmann <teichmann@intevation.de> parents: 978 diff changeset	117 SendJSON(rw, http.StatusCreated, &result)
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	118 }

Mercurial > gemma

annotate pkg/controllers/srimports.go @ 1221:c193649d4f11