gemma: pkg/imports/scheduled.go annotate

annotate pkg/imports/scheduled.go @ 1798:40cbfd268aa9

Row level security for import jobs Add imports to database as the actual login user. sys_admin then only needs to read and update (state) import job data. The way the policies for import_logs and track_imports are implemented has the side effect, that a sys_admin can do anything with all of them. More fine-grainded policies and changes to Go code are needed, if this is a concern.

author	Tom Gottfried <tom@intevation.de>
date	Mon, 14 Jan 2019 17:43:28 +0100
parents	614c6c766691
children	77582da3adb0

rev	line source
1551 d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
1748 c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	17 "context"
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	18 "database/sql"
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	19 "errors"
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	20 "fmt"
1551 d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "log"
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	22
1661 51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1654 diff changeset	23 "gemma.intevation.de/gemma/pkg/common"
1551 d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24 "gemma.intevation.de/gemma/pkg/scheduler"
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 )
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	27 // JobKindSetups maps JobKinds to special setup functions.
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	28 var JobKindSetups = map[JobKind]func(*IDConfig) (interface{}, error){
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	29
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	30 GMJobKind: func(cfg *IDConfig) (interface{}, error) {
1665 da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	31 log.Println("info: schedule 'gm' import")
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	32 insecure := cfg.Attributes.Bool("insecure")
1665 da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	33 return &GaugeMeasurement{
da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	34 URL: *cfg.URL,
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	35 Insecure: insecure,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	36 }, nil
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	37 },
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	38
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	39 FAJobKind: func(cfg *IDConfig) (interface{}, error) {
1665 da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	40 log.Println("info: schedule 'fa' import")
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	41 insecure := cfg.Attributes.Bool("insecure")
1665 da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	42 return &FairwayAvailability{
da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	43 URL: *cfg.URL,
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	44 Insecure: insecure,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	45 }, nil
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	46 },
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	47
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	48 BNJobKind: func(cfg *IDConfig) (interface{}, error) {
1666 56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1665 diff changeset	49 log.Println("info: schedule 'bn' import")
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	50 insecure := cfg.Attributes.Bool("insecure")
1666 56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1665 diff changeset	51 return &Bottleneck{
56b29406a163 Bottleneck import: Made schedulable, too. Even if it's not required. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1665 diff changeset	52 URL: *cfg.URL,
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	53 Insecure: insecure,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	54 }, nil
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	55 },
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	56
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	57 WXJobKind: func(cfg *IDConfig) (interface{}, error) {
1678 8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1666 diff changeset	58 log.Println("info: schedule 'wx' import")
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	59 ft, found := cfg.Attributes.Get("feature-type")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	60 if !found {
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	61 return nil, errors.New("cannot find 'feature-type' attribute")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	62 }
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	63 sb, found := cfg.Attributes.Get("sort-by")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	64 if !found {
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	65 return nil, errors.New("cannot find 'sort-by' attribute")
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	66 }
1678 8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1666 diff changeset	67 return &WaterwayAxis{
8fdb57173e3f Waterway axis import: Made schedulable, too. TODO: Remove hard-coded parameters. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1666 diff changeset	68 URL: *cfg.URL,
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	69 FeatureType: ft,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	70 SortBy: sb,
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	71 }, nil
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	72 },
1785 614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	73
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	74 WAJobKind: func(cfg *IDConfig) (interface{}, error) {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	75 log.Println("info: schedule 'wa' import")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	76 ft, found := cfg.Attributes.Get("feature-type")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	77 if !found {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	78 return nil, errors.New("cannot find 'feature-type' attribute")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	79 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	80 sb, found := cfg.Attributes.Get("sort-by")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	81 if !found {
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	82 return nil, errors.New("cannot find 'sort-by' attribute")
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	83 }
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	84 return &WaterwayArea{
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	85 URL: *cfg.URL,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	86 FeatureType: ft,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	87 SortBy: sb,
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	88 }, nil
614c6c766691 Waterway area import: Implemented. Sascha L. Teichmann <teichmann@intevation.de> parents: 1754 diff changeset	89 },
1551 d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	90 }
d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	91
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	92 func init() {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	93 run := func(cfgID int64) {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	94 jobID, err := RunConfiguredImport(cfgID)
1665 da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	95 if err != nil {
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	96 log.Printf("error: running scheduled import failed: %v\n", err)
1705 dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	97 return
dcbe2a7dc532 Scheduled imports: Take extra configuration parameters from attributes table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1678 diff changeset	98 }
1665 da0d1a19ebe6 Fairway availability import: Made schedulable, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1661 diff changeset	99 log.Printf("info: added import #%d to queue\n", jobID)
1661 51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1654 diff changeset	100 }
51a0ba4ede41 Scheduled imports: Made gauge measurement import able to be scheduled. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1654 diff changeset	101
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	102 for kind := range JobKindSetups {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	103 scheduler.RegisterAction(string(kind), run)
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	104 }
1551 d9eba69f6515 Registered a dummy callback function for scheduled fairway imports in the scheduler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	105 }
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	106
1748 c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	107 // RunConfiguredImportContext runs an import configured from the database.
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	108 func RunConfiguredImportContext(ctx context.Context, conn *sql.Conn, id int64) (int64, error) {
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	109 cfg, err := LoadIDConfigContext(ctx, conn, id)
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	110 return runConfiguredImport(id, cfg, err)
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	111 }
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	112
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	113 // RunConfiguredImport runs an import configured from the database.
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	114 func RunConfiguredImport(id int64) (int64, error) {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	115 cfg, err := loadIDConfig(id)
1748 c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	116 return runConfiguredImport(id, cfg, err)
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	117 }
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	118
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	119 func runConfiguredImport(id int64, cfg *IDConfig, err error) (int64, error) {
c11807dc6940 Comfigured imports: Run enqueing of configured imports if triggered by REST with the database connection of the logged in user. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1746 diff changeset	120
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	121 if err != nil {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	122 return 0, err
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	123 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	124 if cfg == nil {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	125 return 0, fmt.Errorf("no config found for id %d.\n", id)
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	126 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	127 if cfg.URL == nil {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	128 return 0, errors.New("error: No URL specified")
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	129 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	130
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	131 kind := JobKind(cfg.Kind)
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	132
1746 09f51f2b3b0d Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1729 diff changeset	133 setup := JobKindSetups[kind]
09f51f2b3b0d Cosmetics. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1729 diff changeset	134 if setup == nil {
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	135 return 0, fmt.Errorf("unknown job kind: %s", cfg.Kind)
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	136 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	137
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	138 what, err := setup(cfg)
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	139 if err != nil {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	140 return 0, err
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	141 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	142
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	143 var serialized string
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	144 if serialized, err = common.ToJSONString(what); err != nil {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	145 return 0, err
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	146 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	147
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	148 due, _ := cfg.Attributes.Time("due")
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	149
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	150 retries, found := cfg.Attributes.Int("retries")
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	151 if !found {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	152 retries = -1
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	153 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	154
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	155 var jobID int64
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	156 if jobID, err = AddJob(
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	157 kind,
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	158 due, retries,
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	159 cfg.User,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1748 diff changeset	160 cfg.SendEMail,
1729 74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	161 serialized,
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	162 ); err != nil {
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	163 return 0, err
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	164 }
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	165
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	166 return jobID, nil
74f7d4c531bc Scheduled imports: Refactored calling of database configured import to be reusable elsewhere. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	167 }

Mercurial > gemma

annotate pkg/imports/scheduled.go @ 1798:40cbfd268aa9