gemma: pkg/imports/queue.go annotate

annotate pkg/imports/queue.go @ 5111:90b0a14dd58b queued-stage-done

Enable jobs to be removed by the import queue.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Wed, 25 Mar 2020 16:51:45 +0100
parents	4dc2e6dc6c7d
children	4c113aa9a521

rev	line source
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	3 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	6 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	10 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	11 // Author(s):
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1010 diff changeset	13
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package imports
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 "context"
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	19 "encoding/json"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	20 "fmt"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	21 "log"
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	22 "runtime/debug"
3246 64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	23 "sort"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	24 "strings"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "sync"
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	26 "time"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	28 "github.com/jackc/pgx/pgtype"
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	29
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	30 "gemma.intevation.de/gemma/pkg/auth"
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	31 "gemma.intevation.de/gemma/pkg/common"
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	32 "gemma.intevation.de/gemma/pkg/config"
4180 91cb4a7b1b13 Always try to translate to readable error if import failed Tom Gottfried <tom@intevation.de> parents: 4084 diff changeset	33 "gemma.intevation.de/gemma/pkg/pgxutils"
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34 )
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	36 type (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	37 // Feedback is passed to the Do method of a Job to log
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	38 // informations, warnings or errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	39 Feedback interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	40 // Info logs informations.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	41 Info(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	42 // Warn logs warnings.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	43 Warn(fmt string, args ...interface{})
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	44 // Error logs errors.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	45 Error(fmt string, args ...interface{})
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	46 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	47
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	48 // UnchangedError may be issued by Do of a Job to indicate
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	49 // That the database has not changed.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	50 UnchangedError string
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	51
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	52 // Job is the central abstraction of an import job
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	53 // run by the import queue.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	54 Job interface {
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	55 // Do is called to do the actual import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	56 // Bind transactions to ctx and conn, please-
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	57 // id is the number of the import job.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	58 // feedback can be used to log the import process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	59 // If no error is return the import is assumed to
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	60 // be successfull. The non-error return value is
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	61 // serialized as a JSON string into the database as
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	62 // a summary to the import to be used by the review process.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	63 Do(ctx context.Context, id int64, conn *sql.Conn, feedback Feedback) (interface{}, error)
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	64 // CleanUp is called to clean up ressources hold by the import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	65 // It is called whether the import succeeded or not.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	66 CleanUp() error
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	67 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	68
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	69 FeedbackJob interface {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	70 Job
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	71 CreateFeedback(int64) Feedback
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	72 }
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	73
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	74 // JobKind is the type of an import.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	75 // Choose a unique name for every import.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	76 JobKind string
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	77
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	78 // JobCreator is used to bring a job to life as it is stored
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	79 // in pure meta-data form to the database.
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	80 JobCreator interface {
5104 cb736582e8fc Minimize diff to default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5103 diff changeset	81 // Description is the long name of the import.
cb736582e8fc Minimize diff to default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5103 diff changeset	82 Description() string
cb736582e8fc Minimize diff to default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5103 diff changeset	83 // Create build the actual job.
cb736582e8fc Minimize diff to default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5103 diff changeset	84 Create() Job
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	85 // Depends returns two lists of ressources locked by this type of import.
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	86 // Imports are run concurrently if they have disjoint sets
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	87 // of dependencies.
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	88 // The first list are locked exclusively.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	89 // The second allows multiple read users but only one writing one.
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2872 diff changeset	90 Depends() [2][]string
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	91 // StageDone is called if an import is positively reviewed
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	92 // (state = accepted). This can be used to finalize the imported
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	93 // data to move it e.g from the staging area.
5034 59a99655f34d Added feedback support for StageDone. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5033 diff changeset	94 StageDone(context.Context, *sql.Tx, int64, Feedback) error
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	95 // AutoAccept indicates that imports of this kind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	96 // don't need a review.
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	97 AutoAccept() bool
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	98 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	99
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	100 JobRemover interface {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	101 JobCreator
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	102 RemoveJob() bool
5110 4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	103 }
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	104
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	105 idJob struct {
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	106 id int64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	107 kind JobKind
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	108 user string
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	109 waitRetry pgtype.Interval
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	110 trysLeft sql.NullInt64
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	111 sendEmail bool
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	112 data string
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	113 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	114 )
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	115
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	116 const (
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	117 pollDuration = time.Second * 10
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	118 runExclusive = -66666
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	119 )
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	120
5109 c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	121 const (
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	122 ReviewJobSuffix = "#review"
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	123 reviewJobRetrys = 10
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	124 reviewJobWait = time.Minute
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	125 )
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	126
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	127 type importQueue struct {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	128 signalChan chan struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	129 creatorsMu sync.Mutex
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	130 creators map[JobKind]JobCreator
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	131 usedDeps map[string]int
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	132 }
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	133
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	134 var iqueue = importQueue{
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	135 signalChan: make(chan struct{}),
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	136 creators: map[JobKind]JobCreator{},
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	137 usedDeps: map[string]int{},
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	138 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	139
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	140 var (
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	141 // ImportStateNames is a list of the states a job can be in.
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	142 ImportStateNames = []string{
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	143 "queued",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	144 "running",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	145 "failed",
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	146 "unchanged",
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	147 "pending",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	148 "accepted",
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	149 "declined",
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	150 "reviewed",
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	151 }
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	152 )
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1168 diff changeset	153
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	154 const (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	155 queueUser = "sys_admin"
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	156
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	157 reEnqueueRunningSQL = `
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	158 UPDATE import.imports SET
47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	159 state = 'queued'::import_state,
47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	160 changed = CURRENT_TIMESTAMP
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	161 WHERE state = 'running'::import_state`
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	162
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	163 insertJobSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	164 INSERT INTO import.imports (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	165 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	166 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	167 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	168 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	169 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	170 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	171 data
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	172 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	173 $1,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	174 COALESCE($2, CURRENT_TIMESTAMP),
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	175 $3,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	176 $4,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	177 $5,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	178 $6,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	179 $7
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	180 ) RETURNING id`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	181
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	182 selectJobSQL = `
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	183 SELECT
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	184 id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	185 kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	186 trys_left,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	187 retry_wait,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	188 username,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	189 send_email,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	190 data
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	191 FROM import.imports
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	192 WHERE
1718 8ddbedf296d7 Re-scheduled imports: be a bit more tolerant about start time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1708 diff changeset	193 due <= CURRENT_TIMESTAMP + interval '5 seconds' AND
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	194 state = 'queued'::import_state AND enqueued IN (
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	195 SELECT min(enqueued)
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	196 FROM import.imports
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	197 WHERE state = 'queued'::import_state AND
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	198 kind = ANY($1))
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	199 LIMIT 1`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	200
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	201 updateStateSQL = `
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	202 UPDATE import.imports SET
47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	203 state = $1::import_state,
47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	204 changed = CURRENT_TIMESTAMP
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	205 WHERE id = $2`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	206
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	207 updateStateSummarySQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	208 UPDATE import.imports SET
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	209 state = $1::import_state,
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4180 diff changeset	210 changed = CURRENT_TIMESTAMP,
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	211 summary = $2
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	212 WHERE id = $3`
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	213
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	214 deleteJobSQL = `
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	215 DELETE FROM import.imports WHERE id = $1`
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	216
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	217 logMessageSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	218 INSERT INTO import.import_logs (
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	219 import_id,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	220 kind,
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	221 msg
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	222 ) VALUES (
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	223 $1,
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1985 diff changeset	224 $2::log_type,
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	225 $3
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	226 )`
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	227 )
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	228
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	229 func init() {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	230 go iqueue.importLoop()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	231 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	232
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	233 // Error makes UnchangedError an error.
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	234 func (ue UnchangedError) Error() string {
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	235 return string(ue)
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	236 }
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	237
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	238 type reviewedJobCreator struct {
5101 1b0b13e70bc1 Proxy the original job creator directly and not only the dependencies. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5100 diff changeset	239 jobCreator JobCreator
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	240 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	241
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	242 func (*reviewedJobCreator) AutoAccept() bool {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	243 return true
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	244 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	245
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	246 func (*reviewedJobCreator) RemoveJob() bool {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	247 return true
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	248 }
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	249
5101 1b0b13e70bc1 Proxy the original job creator directly and not only the dependencies. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5100 diff changeset	250 func (rjc *reviewedJobCreator) Depends() [2][]string {
5102 8cc5b08ffc2b End endless recursion. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5101 diff changeset	251 return rjc.jobCreator.Depends()
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	252 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	253
5101 1b0b13e70bc1 Proxy the original job creator directly and not only the dependencies. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5100 diff changeset	254 func (rjc *reviewedJobCreator) Description() string {
5109 c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	255 return rjc.Description() + ReviewJobSuffix
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	256 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	257
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	258 func (reviewedJobCreator) StageDone(context.Context, sql.Tx, int64, Feedback) error {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	259 return nil
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	260 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	261
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	262 type reviewedJob struct {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	263 ID int64 `json:"id"`
5103 a11705203f3f Nmae json field in reviewed job correctly. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5102 diff changeset	264 Accepted bool `json:"accepted"`
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	265 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	266
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	267 func (*reviewedJobCreator) Create() Job {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	268 return new(reviewedJob)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	269 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	270
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	271 func (*reviewedJob) CleanUp() error { return nil }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	272
5110 4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	273 func (r *reviewedJob) CreateFeedback(int64) Feedback {
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	274 return logFeedback(r.ID)
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	275 }
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	276
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	277 func (rj *reviewedJob) Do(
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	278 ctx context.Context,
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	279 importID int64,
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	280 conn *sql.Conn,
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	281 feedback Feedback,
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	282 ) (interface{}, error) {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	283
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	284 tx, err := conn.BeginTx(ctx, nil)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	285 if err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	286 return nil, err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	287 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	288 defer tx.Rollback()
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	289
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	290 var signer string
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	291 if err := tx.QueryRowContext(ctx, selectUserSQL, importID).Scan(&signer); err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	292 return nil, err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	293 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	294
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	295 var user, kind string
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	296 if err := tx.QueryRowContext(ctx, selectUserKindSQL, rj.ID).Scan(&user, &kind); err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	297 return nil, err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	298 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	299
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	300 jc := FindJobCreator(JobKind(kind))
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	301 if jc == nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	302 return nil, fmt.Errorf("no job creator found for '%s'", kind)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	303 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	304
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	305 importFeedback := logFeedback(rj.ID)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	306
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	307 if err := auth.RunAs(ctx, user, func(conn *sql.Conn) error {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	308 userTx, err := conn.BeginTx(ctx, nil)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	309 if err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	310 return err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	311 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	312 defer userTx.Rollback()
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	313
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	314 if rj.Accepted {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	315 err = jc.StageDone(ctx, userTx, rj.ID, importFeedback)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	316 } else {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	317 _, err = userTx.ExecContext(ctx, deleteImportDataSQL, rj.ID)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	318 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	319 if err == nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	320 err = userTx.Commit()
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	321 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	322 return err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	323 }); err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	324 return nil, err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	325 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	326
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	327 // Remove the import track
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	328 if _, err := tx.ExecContext(ctx, deleteImportTrackSQL, rj.ID); err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	329 return nil, err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	330 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	331
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	332 var state string
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	333 if rj.Accepted {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	334 state = "accepted"
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	335 } else {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	336 state = "declined"
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	337 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	338
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	339 if _, err := tx.ExecContext(ctx, reviewSQL, state, signer, rj.ID); err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	340 return nil, err
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	341 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	342
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	343 importFeedback.Info("User '%s' %s import %d.", signer, state, rj.ID)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	344
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	345 return nil, tx.Commit()
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	346 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	347
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	348 func (q *importQueue) registerJobCreator(kind JobKind, jc JobCreator) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	349 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	350 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	351 q.creators[kind] = jc
5109 c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	352 q.creators[kind+ReviewJobSuffix] = &reviewedJobCreator{jobCreator: jc}
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	353
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	354 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	355
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	356 // FindJobCreator looks up a JobCreator in the global import queue.
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	357 func FindJobCreator(kind JobKind) JobCreator {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	358 return iqueue.jobCreator(kind)
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	359 }
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1190 diff changeset	360
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	361 // ImportKindNames is a list of the names of the imports the
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	362 // global import queue supports.
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	363 func ImportKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	364 return iqueue.importKindNames()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	365 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	366
3246 64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	367 // LogImportKindNames logs a list of importer types registered
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	368 // to the global import queue.
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	369 func LogImportKindNames() {
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	370 kinds := ImportKindNames()
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	371 sort.Strings(kinds)
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	372 log.Printf("info: registered import kinds: %s",
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	373 strings.Join(kinds, ", "))
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	374 }
64324aaeb1fb Made logging of waht is registered to the scheduler and the import queue more deterministic. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3225 diff changeset	375
1694 4a2fad8f57de Imports: Resolved golint issues unrelated to exported symbols commenting. Sascha L. Teichmann <teichmann@intevation.de> parents: 1657 diff changeset	376 // HasImportKindName checks if the import queue supports a given kind.
1627 b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	377 func HasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	378 return iqueue.hasImportKindName(kind)
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	379 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	380
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	381 //
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	382 func (q *importQueue) hasImportKindName(kind string) bool {
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	383 q.creatorsMu.Lock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	384 defer q.creatorsMu.Unlock()
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	385 return q.creators[JobKind(kind)] != nil
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	386 }
b10aa02d7819 Refactored: Moved REST /api/imports/scheduler to /api/imports/config Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1497 diff changeset	387
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	388 // RegisterJobCreator adds a JobCreator to the global import queue.
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	389 // This a good candidate to be called in a init function for
b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	390 // a particular JobCreator.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	391 func RegisterJobCreator(kind JobKind, jc JobCreator) {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	392 iqueue.registerJobCreator(kind, jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	393 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	394
1495 d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	395 func (q *importQueue) importKindNames() []string {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	396 q.creatorsMu.Lock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	397 defer q.creatorsMu.Unlock()
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	398 names := make([]string, len(q.creators))
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	399 var i int
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	400 for kind := range q.creators {
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	401 names[i] = string(kind)
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	402 i++
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	403 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	404 // XXX: Consider using sort.Strings to make output deterministic.
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	405 return names
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	406 }
d26e3e1fcff1 The global import queue already knows which kinds of imports it supports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1409 diff changeset	407
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	408 func (idj *idJob) nextRetry(feedback Feedback) bool {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	409 switch {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	410 case idj.waitRetry.Status != pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	411 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	412 case idj.waitRetry.Status == pgtype.Present && !idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	413 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	414 case idj.trysLeft.Valid:
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	415 if idj.trysLeft.Int64 < 1 {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	416 feedback.Warn("import should be retried, but no retrys left")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	417 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	418 idj.trysLeft.Int64--
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	419 feedback.Info("import failed but will be retried")
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	420 return true
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	421 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	422 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	423 return false
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	424 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	425
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	426 func (idj *idJob) nextDue() time.Time {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	427 now := time.Now()
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	428 if idj.waitRetry.Status == pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	429 var d time.Duration
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	430 if err := idj.waitRetry.AssignTo(&d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	431 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	432 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	433 now = now.Add(d)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	434 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	435 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	436 return now
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	437 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	438
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	439 func (idj idJob) trysLeftPointer() int {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	440 if !idj.trysLeft.Valid {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	441 return nil
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	442 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	443 t := int(idj.trysLeft.Int64)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	444 return &t
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	445 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	446
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	447 func (idj idJob) waitRetryPointer() time.Duration {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	448 if idj.waitRetry.Status != pgtype.Present {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	449 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	450 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	451 d := new(time.Duration)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	452 if err := idj.waitRetry.AssignTo(d); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	453 log.Printf("error: converting waitRetry failed: %v\n", err)
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	454 return nil
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	455 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	456 return d
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	457 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	458
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	459 func (q *importQueue) lockDependencies(jc JobCreator) {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	460 deps := jc.Depends()
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	461 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	462 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	463 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	464 q.usedDeps[d] = runExclusive
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	465 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	466 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	467 q.usedDeps[d]++
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	468 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	469 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	470
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	471 func (q *importQueue) unlockDependencies(jc JobCreator) {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	472 deps := jc.Depends()
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	473 q.creatorsMu.Lock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	474 defer q.creatorsMu.Unlock()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	475 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	476 q.usedDeps[d] = 0
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	477 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	478 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	479 q.usedDeps[d]--
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	480 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	481 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	482
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	483 func (q *importQueue) jobCreator(kind JobKind) JobCreator {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	484 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	485 defer q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	486 return q.creators[kind]
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	487 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	488
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	489 func (q *importQueue) addJob(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	490 kind JobKind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	491 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	492 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	493 waitRetry *time.Duration,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	494 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	495 sendEmail bool,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	496 data string,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	497 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	498
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	499 var id int64
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	500 if due.IsZero() {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	501 due = time.Now()
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	502 }
4084 350a24c92848 Deliver times from import queue in UTC. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3246 diff changeset	503 due = due.UTC()
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	504
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	505 var tl sql.NullInt64
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	506 if trysLeft != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	507 tl = sql.NullInt64{Int64: int64(*trysLeft), Valid: true}
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	508 }
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	509
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	510 var wr pgtype.Interval
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	511 if waitRetry != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	512 if err := wr.Set(*waitRetry); err != nil {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	513 return 0, err
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	514 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	515 } else {
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	516 wr = pgtype.Interval{Status: pgtype.Null}
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	517 }
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	518
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	519 ctx := context.Background()
1798 40cbfd268aa9 Row level security for import jobs Tom Gottfried <tom@intevation.de> parents: 1760 diff changeset	520 err := auth.RunAs(ctx, user, func(conn *sql.Conn) error {
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	521 return conn.QueryRowContext(
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	522 ctx,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	523 insertJobSQL,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	524 string(kind),
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	525 due,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	526 tl,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	527 &wr,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	528 user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	529 sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	530 data).Scan(&id)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	531 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	532 if err == nil {
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	533 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	534 case q.signalChan <- struct{}{}:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	535 default:
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	536 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	537 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	538 return id, err
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	539 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	540
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	541 // AddJob adds a job to the global import queue to be executed
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	542 // as soon as possible after due.
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	543 // This is gone in a separate Go routine
1497 b41ad15cc55f Backend: Documented the internal API of the global import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	544 // so this will not block.
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	545 func AddJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	546 kind JobKind,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	547 due time.Time,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	548 trysLeft *int,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	549 waitRetry *time.Duration,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	550 user string,
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	551 sendEmail bool,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	552 data string,
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	553 ) (int64, error) {
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	554 return iqueue.addJob(
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	555 kind,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	556 due,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	557 trysLeft,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	558 waitRetry,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	559 user,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	560 sendEmail,
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	561 data)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	562 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	563
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	564 const (
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	565 isPendingSQL = `
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	566 SELECT
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	567 state = 'pending'::import_state,
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	568 kind
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	569 FROM import.imports
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	570 WHERE id = $1`
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	571
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	572 selectKindSQL = `
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	573 SELECT kind from import.imports WHERE ID = $1`
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	574
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	575 selectUserSQL = `
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	576 SELECT username from import.imports WHERE ID = $1`
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	577
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	578 selectUserKindSQL = `
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	579 SELECT username, kind from import.imports WHERE ID = $1`
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	580
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	581 reviewSQL = `
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	582 UPDATE import.imports SET
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	583 state = $1::import_state,
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	584 changed = CURRENT_TIMESTAMP,
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	585 signer = $2
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	586 WHERE id = $3`
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	587
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	588 deleteImportDataSQL = `SELECT import.del_import($1)`
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	589
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	590 deleteImportTrackSQL = `
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	591 DELETE FROM import.track_imports WHERE import_id = $1`
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	592 )
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	593
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	594 func (q *importQueue) decideImportTx(
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	595 ctx context.Context,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	596 tx *sql.Tx,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	597 id int64,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	598 accepted bool,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	599 reviewer string,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	600 ) error {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	601 var (
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	602 pending bool
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	603 kind string
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	604 )
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	605
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	606 switch err := tx.QueryRowContext(ctx, isPendingSQL, id).Scan(&pending, &kind); {
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	607 case err == sql.ErrNoRows:
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	608 return fmt.Errorf("cannot find import #%d", id)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	609 case err != nil:
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	610 return err
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	611 case !pending:
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	612 return fmt.Errorf("#%d is not pending", id)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	613 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	614
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	615 jc := q.jobCreator(JobKind(kind))
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	616 if jc == nil {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	617 return fmt.Errorf("no job creator for kind '%s'", kind)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	618 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	619
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	620 r := &reviewedJob{
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	621 ID: id,
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	622 Accepted: accepted,
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	623 }
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	624 serialized, err := common.ToJSONString(r)
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	625 if err != nil {
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	626 return err
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	627 }
5109 c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	628
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	629 // Try a little harder to persist the decision.
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	630 trys := reviewJobRetrys
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	631 wait := reviewJobWait
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	632
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	633 rID, err := q.addJob(
5109 c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	634 JobKind(kind+ReviewJobSuffix),
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	635 time.Now(),
5109 c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	636 &trys,
c0ceec7e6f85 Define constants for review jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5104 diff changeset	637 &wait,
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	638 reviewer,
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	639 false,
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	640 serialized)
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	641 log.Printf("info: add review job %d\n", rID)
3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	642 if err != nil {
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	643 return err
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	644 }
5099 3cd736acbad3 First version of a reviewed job. I bet it does not work. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5034 diff changeset	645 _, err = tx.ExecContext(ctx, updateStateSQL, "reviewed", id)
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	646 return err
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	647 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	648
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	649 func (q *importQueue) decideImport(
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	650 ctx context.Context,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	651 id int64,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	652 accepted bool,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	653 reviewer string,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	654 ) error {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	655 if ctx == nil {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	656 ctx = context.Background()
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	657 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	658
5033 13d9820c1ea4 Run review decisions SQLs as reviewer instead of queue user to use the rls policies. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5029 diff changeset	659 return auth.RunAs(ctx, reviewer, func(conn *sql.Conn) error {
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	660 tx, err := conn.BeginTx(ctx, nil)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	661 if err != nil {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	662 return err
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	663 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	664 defer tx.Rollback()
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	665 err = q.decideImportTx(ctx, tx, id, accepted, reviewer)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	666 if err == nil {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	667 err = tx.Commit()
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	668 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	669 return err
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	670 })
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	671 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	672
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	673 func DecideImport(
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	674 ctx context.Context,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	675 id int64,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	676 accepted bool,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	677 reviewer string,
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	678 ) error {
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	679 return iqueue.decideImport(ctx, id, accepted, reviewer)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	680 }
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	681
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	682 type logFeedback int64
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	683
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	684 func (lf logFeedback) log(kind, format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	685 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	686 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	687 _, err := conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	688 ctx, logMessageSQL, int64(lf), kind, fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	689 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	690 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	691 if err != nil {
1760 22148eb0f986 More on harmonizing logging. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1758 diff changeset	692 log.Printf("error: logging failed: %v\n", err)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	693 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	694 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	695
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	696 func (lf logFeedback) Info(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	697 lf.log("info", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	698 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	699
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	700 func (lf logFeedback) Warn(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	701 lf.log("warn", format, args...)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	702 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	703
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	704 func (lf logFeedback) Error(format string, args ...interface{}) {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	705 lf.log("error", format, args...)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	706 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	707
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	708 func survive(fn func() error) func() error {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	709 return func() (err error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	710 defer func() {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	711 if p := recover(); p != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	712 err = fmt.Errorf("%v: %s", p, string(debug.Stack()))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	713 }
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	714 }()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	715 return fn()
992 a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	716 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	717 }
a978b2b26a88 Run do and cleanup of import jobs in own go routines with crash handler. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	718
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	719 func reEnqueueRunning() error {
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	720 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	721 return auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	722 _, err := conn.ExecContext(ctx, reEnqueueRunningSQL)
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	723 return err
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	724 })
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	725 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	726
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	727 func (q importQueue) fetchJob() (idJob, error) {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	728
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	729 var which []string
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	730
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	731 q.creatorsMu.Lock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	732 nextCreator:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	733 for kind, jc := range q.creators {
3225 1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	734 deps := jc.Depends()
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	735 for _, d := range deps[0] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	736 if q.usedDeps[d] != 0 {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	737 continue nextCreator
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	738 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	739 }
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	740 for _, d := range deps[1] {
1a0985083c06 Import queue: Fixed exclusive writers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3221 diff changeset	741 if q.usedDeps[d] == runExclusive {
3221 899914a18d7e Import queue: Implemented multiple reader / one writer strategy when locking ressources. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3219 diff changeset	742 continue nextCreator
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	743 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	744 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	745 which = append(which, string(kind))
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	746 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	747 q.creatorsMu.Unlock()
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	748
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	749 if len(which) == 0 {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	750 return nil, sql.ErrNoRows
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	751 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	752
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	753 var kinds pgtype.TextArray
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	754 if err := kinds.Set(which); err != nil {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	755 return nil, err
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	756 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	757
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	758 var ji idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	759 ctx := context.Background()
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	760 err := auth.RunAs(ctx, queueUser, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	761 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	762 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	763 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	764 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	765 defer tx.Rollback()
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	766 if err = tx.QueryRowContext(ctx, selectJobSQL, &kinds).Scan(
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	767 &ji.id,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	768 &ji.kind,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	769 &ji.trysLeft,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	770 &ji.waitRetry,
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	771 &ji.user,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	772 &ji.sendEmail,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	773 &ji.data,
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	774 ); err != nil {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	775 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	776 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	777 _, err = tx.ExecContext(ctx, updateStateSQL, "running", ji.id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	778 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	779 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	780 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	781 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	782 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	783 switch {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	784 case err == sql.ErrNoRows:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	785 return nil, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	786 case err != nil:
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	787 return nil, err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	788 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	789 return &ji, nil
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	790 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	791
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	792 func tryHardToStoreState(ctx context.Context, fn func(*sql.Conn) error) error {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	793 // As it is important to keep the persistent model
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	794 // in sync with the in-memory model try harder to store
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	795 // the state.
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	796 const maxTries = 10
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	797 var sleep = time.Second
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	798
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	799 for try := 1; ; try++ {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	800 var err error
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	801 if err = auth.RunAs(ctx, queueUser, fn); err == nil \|\| try == maxTries {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	802 return err
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	803 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	804 log.Printf("warn: [try %d/%d] Storing state failed: %v (try again in %s).\n",
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	805 try, maxTries, err, sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	806
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	807 time.Sleep(sleep)
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	808 if sleep < time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	809 if sleep *= 2; sleep > time.Minute {
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	810 sleep = time.Minute
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	811 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	812 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	813 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	814 }
bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	815
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	816 func updateStateSummary(
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	817 ctx context.Context,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	818 id int64,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	819 state string,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	820 summary interface{},
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	821 ) error {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	822 var s sql.NullString
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	823 if summary != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	824 var b strings.Builder
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	825 if err := json.NewEncoder(&b).Encode(summary); err != nil {
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	826 return err
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	827 }
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	828 s = sql.NullString{String: b.String(), Valid: true}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	829 }
2823 b150e5b37afe Import queue: As it is important to keep the in-memory and the persistent model in sync try harder to store the final state change if it fails. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2801 diff changeset	830
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	831 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	832 _, err := conn.ExecContext(ctx, updateStateSummarySQL, state, s, id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	833 return err
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	834 })
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	835 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	836
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	837 func deleteJob(ctx context.Context, id int64) error {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	838 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	839 _, err := conn.ExecContext(ctx, deleteJobSQL, id)
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	840 return err
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	841 })
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	842 }
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	843
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	844 func errorAndFail(id int64, format string, args ...interface{}) error {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	845 ctx := context.Background()
2872 bfea3f80ca9a Import queue: Extend the idea of changeset 2886:b150e5b37afe to some error cases that might happen before the storing of the after import state. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2836 diff changeset	846 return tryHardToStoreState(ctx, func(conn *sql.Conn) error {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	847 tx, err := conn.BeginTx(ctx, nil)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	848 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	849 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	850 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	851 defer tx.Rollback()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	852 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	853 ctx, logMessageSQL, id, "error", fmt.Sprintf(format, args...))
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	854 if err != nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	855 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	856 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	857 _, err = conn.ExecContext(
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	858 ctx, updateStateSQL, "failed", id)
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	859 if err == nil {
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	860 err = tx.Commit()
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	861 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	862 return err
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	863 })
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	864 }
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	865
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	866 func (q *importQueue) importLoop() {
1000 14425e35e3c2 Wait with start of import queue until configuration is fully loaded. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 998 diff changeset	867 config.WaitReady()
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	868 // re-enqueue the jobs that are in state running.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	869 // They where in progess when the server went down.
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	870 if err := reEnqueueRunning(); err != nil {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	871 log.Printf("error: re-enqueuing failed: %v", err)
1005 fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	872 }
fcf016ebdef4 Re-enqueue import jobs in state running if the the gemma server starts. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1003 diff changeset	873
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	874 for {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	875 var idj *idJob
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	876 var err error
75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	877
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	878 for {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	879 if idj, err = q.fetchJob(); err != nil && err != sql.ErrNoRows {
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	880 log.Printf("error: db: %v\n", err)
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	881 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	882 if idj != nil {
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	883 break
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	884 }
d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	885 select {
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	886 case <-q.signalChan:
1003 d789f19877f4 Do not rely on internal gemma state only. Instead poll every 10 seconds for new import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1000 diff changeset	887 case <-time.After(pollDuration):
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	888 }
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	889 }
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	890
1751 c3a6aaf926c3 Import queue: Harmonized logging output. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1740 diff changeset	891 log.Printf("info: starting import #%d\n", idj.id)
987 3841509f6e9e Store job id alongside to job in job queue. Sascha L. Teichmann <teichmann@intevation.de> parents: 979 diff changeset	892
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	893 jc := q.jobCreator(idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	894 if jc == nil {
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	895 errorAndFail(idj.id, "no creator for kind '%s' found", idj.kind)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	896 continue
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	897 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 987 diff changeset	898
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	899 // Lock dependencies.
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	900 q.lockDependencies(jc)
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 989 diff changeset	901
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	902 go func(jc JobCreator, idj *idJob) {
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	903
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	904 // Unlock the dependencies.
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	905 defer func() {
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	906 q.unlockDependencies(jc)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	907 select {
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	908 case q.signalChan <- struct{}{}:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	909 default:
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	910 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	911 }()
1010 8f23ec811afb Fixed and harmonized wording in importer queue a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1005 diff changeset	912
5100 d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	913 job := jc.Create()
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	914 if err := common.FromJSONString(idj.data, job); err != nil {
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	915 errorAndFail(idj.id, "failed to create job for import #%d: %v",
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	916 idj.id, err)
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	917 return
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	918 }
d3a24152b0be Register a kind#rev job factory for any job factory. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5099 diff changeset	919
5110 4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	920 var feedback Feedback
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	921 if fc, ok := job.(FeedbackJob); ok {
5110 4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	922 feedback = fc.CreateFeedback(idj.id)
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	923 } else {
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	924 feedback = logFeedback(idj.id)
4dc2e6dc6c7d Redirect logging of review job to original import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	925 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	926
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1136 diff changeset	927 feedback.Info("import #%d started", idj.id)
998 75e65599ea52 Persist job queue in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 992 diff changeset	928
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	929 ctx := context.Background()
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	930 var summary interface{}
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	931
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	932 errDo := survive(func() error {
1327 cabf4789e02b To make golint happier made context.Context to be the first argument of auth.RunAs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	933 return auth.RunAs(ctx, idj.user,
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	934 func(conn *sql.Conn) error {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	935 var err error
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	936 summary, err = job.Do(ctx, idj.id, conn, feedback)
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1328 diff changeset	937 return err
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	938 })
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	939 })()
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	940
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	941 var unchanged, retry bool
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	942 if v, ok := errDo.(UnchangedError); ok {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	943 feedback.Info("unchanged: %s", v.Error())
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	944 unchanged = true
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	945 } else if errDo != nil {
4180 91cb4a7b1b13 Always try to translate to readable error if import failed Tom Gottfried <tom@intevation.de> parents: 4084 diff changeset	946 feedback.Error("error in import: %v",
91cb4a7b1b13 Always try to translate to readable error if import failed Tom Gottfried <tom@intevation.de> parents: 4084 diff changeset	947 pgxutils.ReadableError{Err: errDo})
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	948 retry = idj.nextRetry(feedback)
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	949 }
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	950
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	951 var errCleanup error
2801 054f5d61452d Import queue: Cleanup up debris after an import in all cases if we are not going to retry it. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2187 diff changeset	952 if !retry { // cleanup debris
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	953 if errCleanup = survive(job.CleanUp)(); errCleanup != nil {
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	954 feedback.Error("error cleanup: %v", errCleanup)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	955 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	956 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	957
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	958 var remove bool
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	959 if remover, ok := jc.(JobRemover); ok {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	960 remove = remover.RemoveJob()
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	961 }
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	962
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	963 var state string
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	964 switch {
1975 d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	965 case unchanged:
d966f03ea819 Imports: Added the new state 'unchanged' which can be issued by the imports to indicate that the database is not modified by the particular imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1798 diff changeset	966 state = "unchanged"
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	967 case errDo != nil \|\| errCleanup != nil:
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	968 state = "failed"
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	969 case jc.AutoAccept():
1642 49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	970 state = "accepted"
49c04bb64e0a Import queue: Implemented auto-accept and email sending. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1627 diff changeset	971 default:
1190 e3de65179889 The imort queue has now six states: Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	972 state = "pending"
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	973 }
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	974 if !remove {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	975 if err := updateStateSummary(ctx, idj.id, state, summary); err != nil {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	976 log.Printf("error: setting state of job %d failed: %v\n", idj.id, err)
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	977 }
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	978 log.Printf("info: import #%d finished: %s\n", idj.id, state)
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	979 }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	980 if idj.sendEmail {
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	981 go sendNotificationMail(idj.user, jc.Description(), state, idj.id)
a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1642 diff changeset	982 }
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	983
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	984 switch {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	985 case retry:
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	986 nid, err := q.addJob(
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	987 idj.kind,
1985 8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	988 idj.nextDue(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	989 idj.trysLeftPointer(),
8eeb0b5eb340 Imports: Made retries and the waiting between the attempts configurable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1975 diff changeset	990 idj.waitRetryPointer(),
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1751 diff changeset	991 idj.user, idj.sendEmail,
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	992 idj.data)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	993 if err != nil {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	994 log.Printf("error: retry enqueue failed: %v\n", err)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	995 } else {
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	996 log.Printf("info: re-enqueued job with id %d\n", nid)
49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	997 }
5111 90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	998 case remove:
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	999 if err := deleteJob(ctx, idj.id); err != nil {
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	1000 log.Printf("error: deleting job %d failed: %v\n", idj.id, err)
90b0a14dd58b Enable jobs to be removed by the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5110 diff changeset	1001 }
1708 49e047c2106e Imports: Made imports re-runnable if they fail. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1694 diff changeset	1002 }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	1003 }(jc, idj)
958 2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1004 }
2818ad6c7d32 Started with import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1005 }

Mercurial > gemma

annotate pkg/imports/queue.go @ 5111:90b0a14dd58b queued-stage-done