gemma: pkg/imports/queue.go annotate

annotate pkg/imports/queue.go @ 3225:1a0985083c06

Import queue: Fixed exclusive writers.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 09 May 2019 16:09:46 +0200
parents	899914a18d7e
children	64324aaeb1fb

rev	line source
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	3 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	6 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	10 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	11 // Author(s):
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	13
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	19 "encoding/json"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	20 "fmt"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "log"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	22 "runtime/debug"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	23 "strings"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24 "sync"
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	25 "time"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	27 "github.com/jackc/pgx/pgtype"
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	28
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	29 "gemma.intevation.de/gemma/pkg/auth"
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	30 "gemma.intevation.de/gemma/pkg/common"
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	31 "gemma.intevation.de/gemma/pkg/config"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 )
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	34 type (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	35 // Feedback is passed to the Do method of a Job to log
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	36 // informations, warnings or errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	37 Feedback interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	38 // Info logs informations.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	39 Info(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	40 // Warn logs warnings.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	41 Warn(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	42 // Error logs errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	43 Error(fmt string, args ...interface{})
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	44 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	45
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	46 // UnchangedError may be issued by Do of a Job to indicate
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	47 // That the database has not changed.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	48 UnchangedError string
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	49
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	50 // Job is the central abstraction of an import job
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	51 // run by the import queue.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	52 Job interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	53 // Do is called to do the actual import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	54 // Bind transactions to ctx and conn, please-
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	55 // id is the number of the import job.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	56 // feedback can be used to log the import process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	57 // If no error is return the import is assumed to
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	58 // be successfull. The non-error return value is
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	59 // serialized as a JSON string into the database as
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	60 // a summary to the import to be used by the review process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	61 Do(ctx context.Context, id int64, conn *sql.Conn, feedback Feedback) (interface{}, error)
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	62 // CleanUp is called to clean up ressources hold by the import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	63 // It is called whether the import succeeded or not.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	64 CleanUp() error
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	65 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	66
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	67 // JobKind is the type of an import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	68 // Choose a unique name for every import.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	69 JobKind string
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	70
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	71 // JobCreator is used to bring a job to life as it is stored
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	72 // in pure meta-data form to the database.
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	73 JobCreator interface {
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	74 // Description is the long name of the import.
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	75 Description() string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	76 // Create build the actual job.
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	77 Create() Job
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	78 // Depends returns two lists of ressources locked by this type of import.
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	79 // Imports are run concurrently if they have disjoint sets
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	80 // of dependencies.
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	81 // The first list are locked exclusively.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	82 // The second allows multiple read users but only one writing one.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	83 Depends() [2][]string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	84 // StageDone is called if an import is positively reviewed
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	85 // (state = accepted). This can be used to finalize the imported
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	86 // data to move it e.g from the staging area.
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1327 diff changeset	87 StageDone(context.Context, *sql.Tx, int64) error
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	88 // AutoAccept indicates that imports of this kind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	89 // don't need a review.
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	90 AutoAccept() bool
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	91 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	92
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	93 idJob struct {
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	94 id int64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	95 kind JobKind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	96 user string
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	97 waitRetry pgtype.Interval
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	98 trysLeft sql.NullInt64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	99 sendEmail bool
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	100 data string
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	101 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	102 )
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	103
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	104 const (
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	105 pollDuration = time.Second * 10
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	106 runExclusive = -66666
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	107 )
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	108
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	109 type importQueue struct {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	110 signalChan chan struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	111 creatorsMu sync.Mutex
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	112 creators map[JobKind]JobCreator
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	113 usedDeps map[string]int
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	114 }
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	115
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	116 var iqueue = importQueue{
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	117 signalChan: make(chan struct{}),
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	118 creators: map[JobKind]JobCreator{},
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	119 usedDeps: map[string]int{},
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	120 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	121
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	122 var (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	123 // ImportStateNames is a list of the states a job can be in.
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	124 ImportStateNames = []string{
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	125 "queued",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	126 "running",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	127 "failed",
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	128 "unchanged",
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	129 "pending",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	130 "accepted",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	131 "declined",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	132 }
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	133 )
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	134
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	135 const (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	136 queueUser = "sys_admin"
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	137
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	138 reEnqueueRunningSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	139 UPDATE import.imports SET state = 'queued'::import_state
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	140 WHERE state = 'running'::import_state`
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	141
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	142 insertJobSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	143 INSERT INTO import.imports (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	144 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	145 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	146 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	147 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	148 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	149 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	150 data
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	151 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	152 $1,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	153 COALESCE($2, CURRENT_TIMESTAMP),
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	154 $3,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	155 $4,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	156 $5,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	157 $6,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	158 $7
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	159 ) RETURNING id`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	160
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	161 selectJobSQL = `
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	162 SELECT
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	163 id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	164 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	165 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	166 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	167 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	168 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	169 data
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	170 FROM import.imports
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	171 WHERE
1718 8ddbedf296d7 Re-scheduled imports: be a bit more tolerant about start time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	172 due <= CURRENT_TIMESTAMP + interval '5 seconds' AND
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	173 state = 'queued'::import_state AND enqueued IN (
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	174 SELECT min(enqueued)
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	175 FROM import.imports
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	176 WHERE state = 'queued'::import_state AND
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	177 kind = ANY($1))
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	178 LIMIT 1`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	179
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	180 updateStateSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	181 UPDATE import.imports SET state = $1::import_state
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	182 WHERE id = $2`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	183
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	184 updateStateSummarySQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	185 UPDATE import.imports SET
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	186 state = $1::import_state,
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	187 summary = $2
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	188 WHERE id = $3`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	189
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	190 logMessageSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	191 INSERT INTO import.import_logs (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	192 import_id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	193 kind,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	194 msg
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	195 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	196 $1,
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	197 $2::log_type,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	198 $3
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	199 )`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	200 )
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	201
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	202 func init() {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	203 go iqueue.importLoop()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	204 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	205
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	206 // Error makes UnchangedError an error.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	207 func (ue UnchangedError) Error() string {
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	208 return string(ue)
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	209 }
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	210
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	211 func (q *importQueue) registerJobCreator(kind JobKind, jc JobCreator) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	212 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	213 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	214 q.creators[kind] = jc
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	215 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	216
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	217 // FindJobCreator looks up a JobCreator in the global import queue.
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	218 func FindJobCreator(kind JobKind) JobCreator {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	219 return iqueue.jobCreator(kind)
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	220 }
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	221
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	222 // ImportKindNames is a list of the names of the imports the
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	223 // global import queue supports.
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	224 func ImportKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	225 return iqueue.importKindNames()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	226 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	227
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1657 diff changeset	228 // HasImportKindName checks if the import queue supports a given kind.
1627 b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	229 func HasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	230 return iqueue.hasImportKindName(kind)
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	231 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	232
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	233 //
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	234 func (q *importQueue) hasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	235 q.creatorsMu.Lock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	236 defer q.creatorsMu.Unlock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	237 return q.creators[JobKind(kind)] != nil
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	238 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	239
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	240 // RegisterJobCreator adds a JobCreator to the global import queue.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	241 // This a good candidate to be called in a init function for
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	242 // a particular JobCreator.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	243 func RegisterJobCreator(kind JobKind, jc JobCreator) {
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	244 log.Printf("info: register import job creator for kind '%s'\n", kind)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	245 iqueue.registerJobCreator(kind, jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	246 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	247
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	248 func (q *importQueue) importKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	249 q.creatorsMu.Lock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	250 defer q.creatorsMu.Unlock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	251 names := make([]string, len(q.creators))
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	252 var i int
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	253 for kind := range q.creators {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	254 names[i] = string(kind)
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	255 i++
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	256 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	257 // XXX: Consider using sort.Strings to make output deterministic.
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	258 return names
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	259 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	260
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	261 func (idj *idJob) nextRetry(feedback Feedback) bool {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	262 switch {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	263 case idj.waitRetry.Status != pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	264 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	265 case idj.waitRetry.Status == pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	266 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	267 case idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	268 if idj.trysLeft.Int64 < 1 {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	269 feedback.Warn("import should be retried, but no retrys left")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	270 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	271 idj.trysLeft.Int64--
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	272 feedback.Info("import failed but will be retried")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	273 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	274 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	275 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	276 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	277 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	278
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	279 func (idj *idJob) nextDue() time.Time {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	280 now := time.Now()
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	281 if idj.waitRetry.Status == pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	282 var d time.Duration
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	283 if err := idj.waitRetry.AssignTo(&d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	284 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	285 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	286 now = now.Add(d)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	287 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	288 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	289 return now
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	290 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	291
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	292 func (idj idJob) trysLeftPointer() int {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	293 if !idj.trysLeft.Valid {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	294 return nil
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	295 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	296 t := int(idj.trysLeft.Int64)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	297 return &t
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	298 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	299
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	300 func (idj idJob) waitRetryPointer() time.Duration {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	301 if idj.waitRetry.Status != pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	302 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	303 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	304 d := new(time.Duration)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	305 if err := idj.waitRetry.AssignTo(d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	306 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	307 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	308 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	309 return d
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	310 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	311
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	312 func (q *importQueue) lockDependencies(jc JobCreator) {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	313 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	314 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	315 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	316 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	317 q.usedDeps[d] = runExclusive
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	318 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	319 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	320 q.usedDeps[d]++
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	321 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	322 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	323
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	324 func (q *importQueue) unlockDependencies(jc JobCreator) {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	325 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	326 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	327 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	328 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	329 q.usedDeps[d] = 0
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	330 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	331 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	332 q.usedDeps[d]--
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	333 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	334 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	335
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	336 func (q *importQueue) jobCreator(kind JobKind) JobCreator {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	337 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	338 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	339 return q.creators[kind]
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	340 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	341
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	342 func (q *importQueue) addJob(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	343 kind JobKind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	344 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	345 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	346 waitRetry *time.Duration,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	347 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	348 sendEmail bool,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	349 data string,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	350 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	351
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	352 var id int64
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	353 if due.IsZero() {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	354 due = time.Now()
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	355 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	356
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	357 var tl sql.NullInt64
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	358 if trysLeft != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	359 tl = sql.NullInt64{Int64: int64(*trysLeft), Valid: true}
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	360 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	361
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	362 var wr pgtype.Interval
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	363 if waitRetry != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	364 if err := wr.Set(*waitRetry); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	365 return 0, err
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	366 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	367 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	368 wr = pgtype.Interval{Status: pgtype.Null}
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	369 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	370
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	371 ctx := context.Background()
1798 40cbfd268aa9 Row level security for import jobs Tom Gottfried <tom@intevation.de> parents: 1760 diff changeset	372 err := auth.RunAs(ctx, user, func(conn *sql.Conn) error {
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	373 return conn.QueryRowContext(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	374 ctx,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	375 insertJobSQL,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	376 string(kind),
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	377 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	378 tl,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	379 &wr,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	380 user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	381 sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	382 data).Scan(&id)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	383 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	384 if err == nil {
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	385 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	386 case q.signalChan <- struct{}{}:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	387 default:
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	388 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	389 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	390 return id, err
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	391 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	392
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	393 // AddJob adds a job to the global import queue to be executed
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	394 // as soon as possible after due.
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	395 // This is gone in a separate Go routine
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	396 // so this will not block.
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	397 func AddJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	398 kind JobKind,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	399 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	400 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	401 waitRetry *time.Duration,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	402 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	403 sendEmail bool,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	404 data string,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	405 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	406 return iqueue.addJob(
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	407 kind,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	408 due,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	409 trysLeft,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	410 waitRetry,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	411 user,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	412 sendEmail,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	413 data)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	414 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	415
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	416 type logFeedback int64
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	417
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	418 func (lf logFeedback) log(kind, format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	419 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	420 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	421 _, err := conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	422 ctx, logMessageSQL, int64(lf), kind, fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	423 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	424 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	425 if err != nil {
1760 22148eb0f986 More on harmonizing logging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1758 diff changeset	426 log.Printf("error: logging failed: %v\n", err)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	427 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	428 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	429
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	430 func (lf logFeedback) Info(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	431 lf.log("info", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	432 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	433
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	434 func (lf logFeedback) Warn(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	435 lf.log("warn", format, args...)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	436 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	437
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	438 func (lf logFeedback) Error(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	439 lf.log("error", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	440 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	441
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	442 func survive(fn func() error) func() error {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	443 return func() (err error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	444 defer func() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	445 if p := recover(); p != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	446 err = fmt.Errorf("%v: %s", p, string(debug.Stack()))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	447 }
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	448 }()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	449 return fn()
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	450 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	451 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	452
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	453 func reEnqueueRunning() error {
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	454 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	455 return auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	456 _, err := conn.ExecContext(ctx, reEnqueueRunningSQL)
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	457 return err
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	458 })
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	459 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	460
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	461 func (q importQueue) fetchJob() (idJob, error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	462
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	463 var which []string
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	464
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	465 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	466 nextCreator:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	467 for kind, jc := range q.creators {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	468 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	469 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	470 if q.usedDeps[d] != 0 {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	471 continue nextCreator
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	472 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	473 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	474 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	475 if q.usedDeps[d] == runExclusive {
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	476 continue nextCreator
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	477 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	478 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	479 which = append(which, string(kind))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	480 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	481 q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	482
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	483 if len(which) == 0 {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	484 return nil, sql.ErrNoRows
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	485 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	486
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	487 var kinds pgtype.TextArray
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	488 if err := kinds.Set(which); err != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	489 return nil, err
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	490 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	491
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	492 var ji idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	493 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	494 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	495 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	496 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	497 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	498 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	499 defer tx.Rollback()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	500 if err = tx.QueryRowContext(ctx, selectJobSQL, &kinds).Scan(
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	501 &ji.id,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	502 &ji.kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	503 &ji.trysLeft,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	504 &ji.waitRetry,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	505 &ji.user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	506 &ji.sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	507 &ji.data,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	508 ); err != nil {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	509 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	510 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	511 _, err = tx.ExecContext(ctx, updateStateSQL, "running", ji.id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	512 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	513 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	514 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	515 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	516 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	517 switch {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	518 case err == sql.ErrNoRows:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	519 return nil, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	520 case err != nil:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	521 return nil, err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	522 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	523 return &ji, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	524 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	525
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	526 func tryHardToStoreState(ctx context.Context, fn func(*sql.Conn) error) error {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	527 // As it is important to keep the persistent model
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	528 // in sync with the in-memory model try harder to store
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	529 // the state.
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	530 const maxTries = 10
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	531 var sleep = time.Second
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	532
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	533 for try := 1; ; try++ {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	534 var err error
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	535 if err = auth.RunAs(ctx, queueUser, fn); err == nil \|\| try == maxTries {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	536 return err
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	537 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	538 log.Printf("warn: [try %d/%d] Storing state failed: %v (try again in %s).\n",
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	539 try, maxTries, err, sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	540
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	541 time.Sleep(sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	542 if sleep < time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	543 if sleep *= 2; sleep > time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	544 sleep = time.Minute
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	545 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	546 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	547 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	548 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	549
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	550 func updateStateSummary(
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	551 ctx context.Context,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	552 id int64,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	553 state string,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	554 summary interface{},
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	555 ) error {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	556 var s sql.NullString
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	557 if summary != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	558 var b strings.Builder
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	559 if err := json.NewEncoder(&b).Encode(summary); err != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	560 return err
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	561 }
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	562 s = sql.NullString{String: b.String(), Valid: true}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	563 }
2823 b150e5b37afe Import queue: As it is important to keep the in-memory and the persistent model in sync try harder to store the final state change if it fails. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2801 diff changeset	564
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	565 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	566 _, err := conn.ExecContext(ctx, updateStateSummarySQL, state, s, id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	567 return err
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	568 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	569 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	570
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	571 func errorAndFail(id int64, format string, args ...interface{}) error {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	572 ctx := context.Background()
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	573 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	574 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	575 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	576 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	577 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	578 defer tx.Rollback()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	579 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	580 ctx, logMessageSQL, id, "error", fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	581 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	582 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	583 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	584 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	585 ctx, updateStateSQL, "failed", id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	586 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	587 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	588 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	589 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	590 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	591 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	592
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	593 func (q *importQueue) importLoop() {
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	594 config.WaitReady()
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	595 // re-enqueue the jobs that are in state running.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	596 // They where in progess when the server went down.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	597 if err := reEnqueueRunning(); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	598 log.Printf("error: re-enqueuing failed: %v", err)
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	599 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	600
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	601 for {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	602 var idj *idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	603 var err error
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	604
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	605 for {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	606 if idj, err = q.fetchJob(); err != nil && err != sql.ErrNoRows {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	607 log.Printf("error: db: %v\n", err)
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	608 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	609 if idj != nil {
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	610 break
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	611 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	612 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	613 case <-q.signalChan:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	614 case <-time.After(pollDuration):
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	615 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	616 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	617
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	618 log.Printf("info: starting import #%d\n", idj.id)
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	619
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	620 jc := q.jobCreator(idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	621 if jc == nil {
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	622 errorAndFail(idj.id, "no creator for kind '%s' found", idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	623 continue
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	624 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	625
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	626 // Lock dependencies.
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	627 q.lockDependencies(jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	628
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	629 go func(jc JobCreator, idj *idJob) {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	630
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	631 // Unlock the dependencies.
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	632 defer func() {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	633 q.unlockDependencies(jc)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	634 select {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	635 case q.signalChan <- struct{}{}:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	636 default:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	637 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	638 }()
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	639
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	640 job := jc.Create()
7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	641 if err := common.FromJSONString(idj.data, job); err != nil {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	642 errorAndFail(idj.id, "failed to create job for import #%d: %v",
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	643 idj.id, err)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	644 return
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	645 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	646
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	647 feedback := logFeedback(idj.id)
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	648
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1136 diff changeset	649 feedback.Info("import #%d started", idj.id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	650
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	651 ctx := context.Background()
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	652 var summary interface{}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	653
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	654 errDo := survive(func() error {
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	655 return auth.RunAs(ctx, idj.user,
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	656 func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	657 var err error
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	658 summary, err = job.Do(ctx, idj.id, conn, feedback)
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	659 return err
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	660 })
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	661 })()
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	662
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	663 var unchanged, retry bool
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	664 if v, ok := errDo.(UnchangedError); ok {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	665 feedback.Info("unchanged: %s", v.Error())
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	666 unchanged = true
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	667 } else if errDo != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	668 feedback.Error("error in import: %v", errDo)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	669 retry = idj.nextRetry(feedback)
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	670 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	671
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	672 var errCleanup error
2801 054f5d61452d Import queue: Cleanup up debris after an import in all cases if we are not going to retry it. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	673 if !retry { // cleanup debris
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	674 if errCleanup = survive(job.CleanUp)(); errCleanup != nil {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	675 feedback.Error("error cleanup: %v", errCleanup)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	676 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	677 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	678
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	679 var state string
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	680 switch {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	681 case unchanged:
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	682 state = "unchanged"
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	683 case errDo != nil \|\| errCleanup != nil:
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	684 state = "failed"
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	685 case jc.AutoAccept():
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	686 state = "accepted"
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	687 default:
1190 e3de65179889 The imort queue has now six states: Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	688 state = "pending"
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	689 }
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	690 if err := updateStateSummary(ctx, idj.id, state, summary); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	691 log.Printf("error: setting state of job %d failed: %v\n", idj.id, err)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	692 }
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	693 log.Printf("info: import #%d finished: %s\n", idj.id, state)
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	694 if idj.sendEmail {
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	695 go sendNotificationMail(idj.user, jc.Description(), state, idj.id)
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	696 }
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	697
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	698 if retry {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	699 nid, err := q.addJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	700 idj.kind,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	701 idj.nextDue(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	702 idj.trysLeftPointer(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	703 idj.waitRetryPointer(),
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	704 idj.user, idj.sendEmail,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	705 idj.data)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	706 if err != nil {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	707 log.Printf("error: retry enqueue failed: %v\n", err)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	708 } else {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	709 log.Printf("info: re-enqueued job with id %d\n", nid)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	710 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	711 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	712 }(jc, idj)
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	713 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	714 }

Mercurial > gemma

annotate pkg/imports/queue.go @ 3225:1a0985083c06