gemma: pkg/controllers/importqueue.go annotate

annotate pkg/controllers/importqueue.go @ 5562:5152b4db40cc

Simplified code of log export a bit.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Sat, 12 Feb 2022 09:54:33 +0100
parents	b91716d2acc6
children	3be1d79ad3a3

rev	line source
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	2 // without warranty, see README.md and license for details.
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	3 //
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	6 //
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	7 // Copyright (C) 2018 by via donau
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	9 // Software engineering by Intevation GmbH
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	10 //
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	11 // Author(s):
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	13
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	14 package controllers
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16 import (
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	17 "context"
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "database/sql"
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	19 "encoding/csv"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1359 diff changeset	20 "encoding/json"
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	21 "fmt"
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	22 "net/http"
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	23 "strconv"
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	24 "strings"
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	25 "sync"
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	26 "time"
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	27
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1359 diff changeset	28 "github.com/gorilla/mux"
5561 b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	29 "github.com/jackc/pgx/pgtype"
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1359 diff changeset	30
1194 7db850de0952 Added a signer who makes the final decison on an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	31 "gemma.intevation.de/gemma/pkg/auth"
4078 80bdcd137a1d Parse timezones from time inputs and send timezones in results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3776 diff changeset	32 "gemma.intevation.de/gemma/pkg/common"
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	33 "gemma.intevation.de/gemma/pkg/imports"
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5153 diff changeset	34 "gemma.intevation.de/gemma/pkg/log"
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	35 "gemma.intevation.de/gemma/pkg/models"
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	36
4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	37 mw "gemma.intevation.de/gemma/pkg/middleware"
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	38 )
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40 const (
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	41 selectImportsCountSQL = `
2682 9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	42 SELECT count(*)
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	43 FROM import.imports
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	44 WHERE
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	45 `
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	46 selectImportsSQL = `
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	47 SELECT
2627 3a242e6aa56d Import log: Add filter for log entries with warnings only: GET /api/imports?warnings=true Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2179 diff changeset	48 imports.id AS id,
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	49 state::varchar,
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	50 enqueued,
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4244 diff changeset	51 changed,
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	52 kind,
1194 7db850de0952 Added a signer who makes the final decison on an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	53 username,
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1359 diff changeset	54 signer,
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	55 summary IS NOT NULL AS has_summary,
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	56 EXISTS(SELECT 1 FROM import.import_logs
4196 948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	57 WHERE kind = 'error'::log_type and import_id = imports.id) AS has_errors,
948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	58 EXISTS(SELECT 1 FROM import.import_logs
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	59 WHERE kind = 'warn'::log_type and import_id = imports.id) AS has_warnings
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1693 diff changeset	60 FROM import.imports
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	61 WHERE
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	62 `
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	63 selectExportSQL = `
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	64 SELECT
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	65 imports.id AS id,
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	66 state::varchar,
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	67 enqueued,
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	68 changed,
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	69 imports.kind,
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	70 username,
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	71 (SELECT country FROM users.list_users lu
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	72 WHERE lu.username = import.imports.username) AS country,
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	73 signer,
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	74 EXISTS(SELECT 1 FROM import.import_logs
5561 b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	75 WHERE kind = 'warn'::log_type and import_id = id) AS has_warnings,
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	76 data,
5561 b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	77 ARRAY(SELECT msg FROM import.import_logs WHERE import_id = id) AS msgs
b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	78 FROM import.imports
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	79 WHERE
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	80 `
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	81 selectEnqueuedSQL = `
2672 b997e1fd1d3d Fixed warning SQL prefix for selection. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2671 diff changeset	82 SELECT enqueued FROM import.imports
b997e1fd1d3d Fixed warning SQL prefix for selection. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2671 diff changeset	83 WHERE
b997e1fd1d3d Fixed warning SQL prefix for selection. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2671 diff changeset	84 `
2795 241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	85 selectImportSummarySQL = `
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	86 SELECT summary, enqueued FROM import.imports WHERE id = $1`
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	87
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	88 selectHasNoRunningImportSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1693 diff changeset	89 SELECT true FROM import.imports
59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1693 diff changeset	90 WHERE id = $1 AND state <> 'running'::import_state`
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	91
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	92 selectImportLogsSQL = `
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	93 SELECT
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	94 time,
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	95 kind::varchar,
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	96 msg
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1693 diff changeset	97 FROM import.import_logs
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	98 WHERE import_id = $1
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	99 ORDER BY time`
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	100
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	101 deleteLogsSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1693 diff changeset	102 DELETE FROM import.import_logs WHERE import_id = $1`
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	103
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	104 deleteImportSQL = `
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1693 diff changeset	105 DELETE FROM import.imports WHERE id = $1`
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	106 )
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	107
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	108 type filledStmt struct {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	109 stmt strings.Builder
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	110 args []interface{}
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	111 }
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	112
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	113 func buildFilters(projection string, req http.Request) (filledStmt, filledStmt, filledStmt, error) {
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	114
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	115 var l, a, b filterAnd
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	116
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	117 var noBefore, noAfter bool
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	118
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	119 cond := func(format string, args ...interface{}) {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	120 term := &filterTerm{format: format, args: args}
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	121 l = append(l, term)
4195 ad13c581de7c Fixed building filters for listing import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4187 diff changeset	122 a = append(a, term)
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	123 b = append(b, term)
2677 fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	124 }
fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	125
5108 89639e02cff0 Do not show internal review jobs to users Tom Gottfried <tom@intevation.de> parents: 5093 diff changeset	126 // Always filter review jobs. They are only for internal use.
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	127 cond(` NOT imports.kind LIKE '%%` + imports.ReviewJobSuffix + `'`)
5108 89639e02cff0 Do not show internal review jobs to users Tom Gottfried <tom@intevation.de> parents: 5093 diff changeset	128
2677 fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	129 if query := req.FormValue("query"); query != "" {
fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	130 query = "%" + query + "%"
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	131 cond(` (imports.kind ILIKE $%d OR username ILIKE $%d OR signer ILIKE $%d OR `+
2677 fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	132 `id IN (SELECT import_id FROM import.import_logs WHERE msg ILIKE $%d)) `,
fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	133 query, query, query, query)
fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	134 }
fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	135
3776 6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	136 if cc := req.FormValue("cc"); cc != "" {
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	137 codes := sliceToTextArray(splitUpper(cc))
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	138 cond(" username IN "+
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	139 "(SELECT username FROM internal.user_profiles "+
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	140 "WHERE country = ANY($%d)) ",
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	141 codes)
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	142 }
6521c962a7b6 Add 'cc' parameter to search imports for a comma separated list of country codes for the the importing person. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3217 diff changeset	143
2667 5ece2c51d1f0 More filter simplification. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2665 diff changeset	144 if st := req.FormValue("states"); st != "" {
5ece2c51d1f0 More filter simplification. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2665 diff changeset	145 states := toTextArray(st, imports.ImportStateNames)
2677 fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	146 cond(" state = ANY($%d) ", states)
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	147 }
3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	148
2667 5ece2c51d1f0 More filter simplification. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2665 diff changeset	149 if ks := req.FormValue("kinds"); ks != "" {
5ece2c51d1f0 More filter simplification. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2665 diff changeset	150 kinds := toTextArray(ks, imports.ImportKindNames())
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	151 cond(" imports.kind = ANY($%d) ", kinds)
1655 34315277f2d6 Added ids filter to importqueue listing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	152 }
34315277f2d6 Added ids filter to importqueue listing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	153
2667 5ece2c51d1f0 More filter simplification. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2665 diff changeset	154 if idss := req.FormValue("ids"); idss != "" {
5ece2c51d1f0 More filter simplification. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2665 diff changeset	155 ids := toInt8Array(idss)
2677 fe93c48f76c9 Add /api/imports?query=TXT to search TXT in kind, username, signer or logs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2673 diff changeset	156 cond(" id = ANY($%d) ", ids)
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	157 }
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	158
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	159 if from := req.FormValue("from"); from != "" {
4078 80bdcd137a1d Parse timezones from time inputs and send timezones in results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3776 diff changeset	160 fromTime, err := common.ParseTime(from)
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	161 if err != nil {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	162 return nil, nil, nil, err
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	163 }
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	164 l = append(l, buildFilterTerm("enqueued >= $%d", fromTime))
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	165 b = append(b, buildFilterTerm("enqueued < $%d", fromTime))
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	166 } else {
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	167 noBefore = true
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	168 }
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	169
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	170 if to := req.FormValue("to"); to != "" {
4078 80bdcd137a1d Parse timezones from time inputs and send timezones in results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3776 diff changeset	171 toTime, err := common.ParseTime(to)
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	172 if err != nil {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	173 return nil, nil, nil, err
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	174 }
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	175 l = append(l, buildFilterTerm("enqueued <= $%d", toTime))
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	176 a = append(a, buildFilterTerm("enqueued > $%d", toTime))
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	177 } else {
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	178 noAfter = true
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	179 }
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	180
2665 fea12fc850d2 Simplified warnings filtering. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2663 diff changeset	181 switch warn := strings.ToLower(req.FormValue("warnings")); warn {
fea12fc850d2 Simplified warnings filtering. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2663 diff changeset	182 case "1", "t", "true":
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	183 cond(` EXISTS(SELECT 1 FROM import.import_logs
65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	184 WHERE kind = 'warn'::log_type and import_id = imports.id)`)
2665 fea12fc850d2 Simplified warnings filtering. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2663 diff changeset	185 }
fea12fc850d2 Simplified warnings filtering. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2663 diff changeset	186
4196 948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	187 switch errors := strings.ToLower(req.FormValue("errors")); errors {
948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	188 case "1", "t", "true":
948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	189 cond(` EXISTS(SELECT 1 FROM import.import_logs
948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	190 WHERE kind = 'error'::log_type and import_id = imports.id)`)
948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	191 }
948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	192
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	193 fl := &filledStmt{}
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	194 fa := &filledStmt{}
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	195 fb := &filledStmt{}
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	196
4187 65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	197 fa.stmt.WriteString(selectEnqueuedSQL)
65a5501dc13d Enable faster access to import logs Tom Gottfried <tom@intevation.de> parents: 4148 diff changeset	198 fb.stmt.WriteString(selectEnqueuedSQL)
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	199
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	200 var counting bool
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	201
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	202 if projection != "" {
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	203 fl.stmt.WriteString(projection)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	204 } else {
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	205 switch count := strings.ToLower(req.FormValue("count")); count {
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	206 case "1", "t", "true":
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	207 counting = true
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	208 fl.stmt.WriteString(selectImportsCountSQL)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	209 default:
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	210 fl.stmt.WriteString(selectImportsSQL)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	211 }
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	212 }
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	213
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	214 if len(l) == 0 {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	215 fl.stmt.WriteString(" TRUE ")
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	216 } else {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	217 l.serialize(&fl.stmt, &fl.args)
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	218 }
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	219
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	220 if len(b) == 0 {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	221 fb.stmt.WriteString(" TRUE ")
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	222 } else {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	223 b.serialize(&fb.stmt, &fb.args)
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	224 }
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	225
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	226 if len(a) == 0 {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	227 fa.stmt.WriteString(" TRUE ")
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	228 } else {
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	229 a.serialize(&fa.stmt, &fa.args)
1655 34315277f2d6 Added ids filter to importqueue listing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	230 }
34315277f2d6 Added ids filter to importqueue listing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1495 diff changeset	231
2682 9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	232 if !counting {
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	233 fl.stmt.WriteString(" ORDER BY enqueued DESC ")
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	234 fa.stmt.WriteString(" ORDER BY enqueued LIMIT 1")
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	235 fb.stmt.WriteString(" ORDER BY enqueued DESC LIMIT 1")
2682 9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	236 }
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	237
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	238 if noBefore {
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	239 fb = nil
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	240 }
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	241 if noAfter {
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	242 fa = nil
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	243 }
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	244
eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	245 return fl, fb, fa, nil
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	246 }
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	247
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	248 func neighbored(ctx context.Context, conn sql.Conn, fb filledStmt) *models.ImportTime {
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	249
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	250 var when time.Time
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	251 err := conn.QueryRowContext(ctx, fb.stmt.String(), fb.args...).Scan(&when)
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	252 switch {
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	253 case err == sql.ErrNoRows:
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	254 return nil
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	255 case err != nil:
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5153 diff changeset	256 log.Warnf("%v\n", err)
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	257 return nil
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	258 }
4084 350a24c92848 Deliver times from import queue in UTC. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4078 diff changeset	259 return &models.ImportTime{Time: when.UTC()}
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	260 }
1189 3d50f558870c REST GET call to /imports now has the ability to be filtered by kinds or states. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1037 diff changeset	261
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	262 func exportImports(rw http.ResponseWriter, req *http.Request) {
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	263
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	264 list, _, _, err := buildFilters(selectExportSQL, req)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	265 if err != nil {
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	266 http.Error(rw, "error: "+err.Error(), http.StatusBadRequest)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	267 return
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	268 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	269
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	270 rw.Header().Add("Content-Type", "text/csv")
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	271 out := csv.NewWriter(rw)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	272
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	273 record := []string{
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	274 "#id",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	275 "#kind",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	276 "#enqueued",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	277 "#changed",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	278 "#user",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	279 "#country",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	280 "#signer",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	281 "#state",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	282 "#warnings",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	283 "#source",
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	284 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	285
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	286 if err := out.Write(record); err != nil {
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	287 // Too late for HTTP status message.
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5153 diff changeset	288 log.Errorf("%v\n", err)
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	289 return
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	290 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	291
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	292 start := time.Now()
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	293
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	294 conn := mw.GetDBConn(req)
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	295 ctx := req.Context()
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	296 var rows *sql.Rows
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	297 if rows, err = conn.QueryContext(ctx, list.stmt.String(), list.args...); err != nil {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5153 diff changeset	298 log.Errorf("%v\n", err)
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	299 return
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	300 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	301 defer rows.Close()
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	302
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	303 // Extract some meta infos from the import.
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	304 type Description interface {
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	305 Description([]string) (string, error)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	306 }
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	307
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	308 type dataset struct {
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	309 id int64
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	310 state string
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	311 enqueued time.Time
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	312 changed time.Time
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	313 kind string
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	314 user string
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	315 country string
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	316 signer sql.NullString
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	317 warnings bool
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	318 data string
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	319 msgs pgtype.TextArray
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	320 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	321
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	322 // Log unsupported description interfaces per kind only once.
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	323 unsupported := make(map[string]bool)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	324
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	325 store := func(ds *dataset) error {
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	326
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	327 var description string
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	328
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	329 // Do some introspection on the job to be more verbose.
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	330 if jc := imports.FindJobCreator(imports.JobKind(ds.kind)); jc != nil {
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	331 job := jc.Create()
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	332 if err := common.FromJSONString(ds.data, job); err != nil {
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	333 log.Errorf("%v\n", err)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	334 } else if desc, ok := job.(Description); ok {
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	335 var ms []string
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	336 if ds.msgs.Status == pgtype.Present {
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	337 if err := ds.msgs.AssignTo(&ms); err != nil {
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	338 return err
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	339 }
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	340 }
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	341 if description, err = desc.Description(ms); err != nil {
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	342 return err
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	343 }
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	344 description = strings.Replace(description, ",", "\|", -1)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	345 } else {
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	346 if !unsupported[ds.kind] {
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	347 unsupported[ds.kind] = true
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	348 log.Debugf("%s: description not supported\n", ds.kind)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	349 }
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	350 }
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	351 }
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	352
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	353 var signer string
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	354 if ds.signer.Valid {
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	355 signer = ds.signer.String
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	356 }
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	357
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	358 record[0] = strconv.FormatInt(ds.id, 10)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	359 record[1] = ds.kind
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	360 record[2] = ds.enqueued.UTC().Format(common.TimeFormat)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	361 record[3] = ds.changed.UTC().Format(common.TimeFormat)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	362 record[4] = ds.user
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	363 record[5] = ds.country
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	364 record[6] = signer
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	365 record[7] = ds.state
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	366 record[8] = strconv.FormatBool(ds.warnings)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	367 record[9] = description
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	368
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	369 return out.Write(record)
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	370 }
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	371
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	372 for rows.Next() {
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	373 var curr dataset
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	374
5561 b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	375 if err := rows.Scan(
5560 f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	376 &curr.id,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	377 &curr.state,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	378 &curr.enqueued,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	379 &curr.changed,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	380 &curr.kind,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	381 &curr.user,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	382 &curr.country,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	383 &curr.signer,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	384 &curr.warnings,
f2204f91d286 Join the log lines of imports to the log exports to recover data from them. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5490 diff changeset	385 &curr.data,
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	386 &curr.msgs,
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	387 ); err != nil {
5561 b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	388 log.Errorf("%v\n", err)
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	389 return
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	390 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	391
5561 b91716d2acc6 Fixed log message analyzing in log export. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5560 diff changeset	392 if err := store(&curr); err != nil {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5153 diff changeset	393 log.Errorf("%v\n", err)
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	394 return
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	395 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	396 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	397
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	398 if err := rows.Err(); err != nil {
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	399 log.Errorf("%v\n", err)
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	400 return
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	401 }
5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	402
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	403 out.Flush()
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	404 if err := out.Error(); err != nil {
5490 5f47eeea988d Use own logging package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5153 diff changeset	405 log.Errorf("%v\n", err)
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	406 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	407
5562 5152b4db40cc Simplified code of log export a bit. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5561 diff changeset	408 log.Debugf("Export took: %v\n", time.Since(start))
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	409 }
1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	410
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	411 func listImports(req *http.Request) (jr mw.JSONResult, err error) {
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	412
3194 eeff2cc4ff9d controllers: re-factored the SQL filter to a tree like structure to be of more general use. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 3190 diff changeset	413 var list, before, after *filledStmt
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	414
4791 1fef4679b07a Added an endpoint GET api/imports/export to export imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4748 diff changeset	415 if list, before, after, err = buildFilters("", req); err != nil {
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	416 return
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	417 }
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	418
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	419 ctx := req.Context()
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	420
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	421 conn := mw.JSONConn(req)
4242 1458c9b0fdaa Made the sql.Conn in function accessible via the context of the request. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4196 diff changeset	422
2682 9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	423 // Fast path for counting
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	424
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	425 switch count := strings.ToLower(req.FormValue("count")); count {
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	426 case "1", "t", "true":
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	427 var count int64
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	428 err = conn.QueryRowContext(ctx, list.stmt.String(), list.args...).Scan(&count)
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	429 switch {
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	430 case err == sql.ErrNoRows:
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	431 count, err = 0, nil
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	432 case err != nil:
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	433 return
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	434 }
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	435 jr = mw.JSONResult{Result: count}
2682 9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	436 return
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	437 }
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	438
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	439 // Generate the list
9a493d27bf3f Added /api/imports?count=true to only count the results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2681 diff changeset	440
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	441 var rows *sql.Rows
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	442 if rows, err = conn.QueryContext(ctx, list.stmt.String(), list.args...); err != nil {
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	443 return
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	444 }
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	445 defer rows.Close()
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	446
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	447 imports := make([]*models.Import, 0, 20)
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	448
2662 f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	449 var signer sql.NullString
1195 486d66a9565c Be aware that the signer is null till the final decision is made about an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1194 diff changeset	450
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	451 for rows.Next() {
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	452 var it models.Import
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	453 var enqueued time.Time
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4244 diff changeset	454 var changed time.Time
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	455 if err = rows.Scan(
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	456 &it.ID,
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	457 &it.State,
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	458 &enqueued,
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4244 diff changeset	459 &changed,
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	460 &it.Kind,
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	461 &it.User,
1195 486d66a9565c Be aware that the signer is null till the final decision is made about an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1194 diff changeset	462 &signer,
2662 f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	463 &it.Summary,
4196 948e312e87bc Add flag and filter for imports having errors Tom Gottfried <tom@intevation.de> parents: 4195 diff changeset	464 &it.Errors,
2179 20d9b71f4125 Import queue listing: Add another column 'warnings' (bool) to listing output which indicates if an import has warnings or not. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1995 diff changeset	465 &it.Warnings,
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	466 ); err != nil {
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	467 return
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	468 }
1195 486d66a9565c Be aware that the signer is null till the final decision is made about an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1194 diff changeset	469 if signer.Valid {
486d66a9565c Be aware that the signer is null till the final decision is made about an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1194 diff changeset	470 it.Signer = signer.String
486d66a9565c Be aware that the signer is null till the final decision is made about an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1194 diff changeset	471 }
4084 350a24c92848 Deliver times from import queue in UTC. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4078 diff changeset	472 it.Enqueued = models.ImportTime{Time: enqueued.UTC()}
4748 47922c1a088d Added a 'changed' column to the import.imports table. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4244 diff changeset	473 it.Changed = models.ImportTime{Time: changed.UTC()}
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	474 imports = append(imports, &it)
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	475 }
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	476
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	477 if err = rows.Err(); err != nil {
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	478 return
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	479 }
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	480
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	481 var prev, next *models.ImportTime
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	482
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	483 if before != nil {
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	484 prev = neighbored(ctx, conn, before)
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	485 }
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	486
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	487 if after != nil {
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	488 next = neighbored(ctx, conn, after)
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	489 }
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	490
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	491 jr = mw.JSONResult{
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	492 Result: struct {
2663 386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	493 Prev *models.ImportTime `json:"prev,omitempty"`
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	494 Next *models.ImportTime `json:"next,omitempty"`
386ff766dfcd Replaced limit/offset filters with from/to as timestamps (format '2006-01-02T15:04:05.000').The output document also contains 'prev' and 'next' fields (if there is any matching data) from the times before and after 'from' and 'to'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2662 diff changeset	495 Imports []*models.Import `json:"imports"`
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	496 }{
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	497 Imports: imports,
2671 8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	498 Prev: prev,
8f3facf902dd Filter prev/next, too. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2670 diff changeset	499 Next: next,
1023 337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	500 },
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	501 }
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	502 return
337a7f4c8a16 Add endpoint to list all or some import jobs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	503 }
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	504
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	505 func importLogs(req *http.Request) (jr mw.JSONResult, err error) {
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	506
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	507 ctx := req.Context()
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	508
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	509 id, _ := strconv.ParseInt(mux.Vars(req)["id"], 10, 64)
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	510
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	511 conn := mw.JSONConn(req)
4242 1458c9b0fdaa Made the sql.Conn in function accessible via the context of the request. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4196 diff changeset	512
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	513 // Check if he have such a import job first.
2662 f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	514 var summary sql.NullString
2795 241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	515 var enqueued time.Time
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	516 err = conn.QueryRowContext(ctx, selectImportSummarySQL, id).Scan(
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	517 &summary,
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	518 &enqueued,
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	519 )
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	520 switch {
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	521 case err == sql.ErrNoRows:
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	522 err = mw.JSONError{
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	523 Code: http.StatusNotFound,
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	524 Message: fmt.Sprintf("Cannot find import #%d.", id),
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	525 }
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	526 return
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	527 case err != nil:
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	528 return
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	529 }
4084 350a24c92848 Deliver times from import queue in UTC. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4078 diff changeset	530 enqueued = enqueued.UTC()
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	531
2662 f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	532 var sum interface{}
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	533 if summary.Valid {
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	534 if err = json.NewDecoder(
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	535 strings.NewReader(summary.String)).Decode(&sum); err != nil {
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	536 return
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	537 }
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	538 }
f90557a8c960 Moved summary from /api/imports overview to /api/imports/{id} details. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2627 diff changeset	539
1031 c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	540 // We have it -> generate log entries.
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	541 var rows *sql.Rows
c0f5dedf5753 Check first if we have a import before generating log import entries list. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1026 diff changeset	542 rows, err = conn.QueryContext(ctx, selectImportLogsSQL, id)
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	543 if err != nil {
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	544 return
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	545 }
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	546 defer rows.Close()
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	547
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	548 entries := make([]*models.ImportLogEntry, 0, 10)
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	549
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	550 for rows.Next() {
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	551 var entry models.ImportLogEntry
4084 350a24c92848 Deliver times from import queue in UTC. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4078 diff changeset	552 var t time.Time
350a24c92848 Deliver times from import queue in UTC. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4078 diff changeset	553 if err = rows.Scan(&t, &entry.Kind, &entry.Message); err != nil {
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	554 return
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	555 }
4148 1b1218bbd414 Made 'go vet' happy with controllers again. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4084 diff changeset	556 entry.Time = models.ImportTime{Time: t.UTC()}
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	557 entries = append(entries, &entry)
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	558 }
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	559
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	560 if err = rows.Err(); err != nil {
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	561 return
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	562 }
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	563
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	564 jr = mw.JSONResult{
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	565 Result: struct {
2795 241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	566 Enqueued models.ImportTime `json:"enqueued"`
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	567 Summary interface{} `json:"summary,omitempty"`
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	568 Entries []*models.ImportLogEntry `json:"entries"`
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	569 }{
3089 813309225e35 Made 'go vet' happy again. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2954 diff changeset	570 Enqueued: models.ImportTime{Time: enqueued},
2795 241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	571 Summary: sum,
241e7f05a538 /api/imports/{id} result now also contains the 'enqueued' time. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2772 diff changeset	572 Entries: entries,
1026 3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	573 },
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	574 }
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	575 return
3de54d7b7d30 Added endpoint to show the detail logs of an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1023 diff changeset	576 }
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	577
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	578 func deleteImport(req *http.Request) (jr mw.JSONResult, err error) {
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	579
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	580 ctx := req.Context()
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	581 id, _ := strconv.ParseInt(mux.Vars(req)["id"], 10, 64)
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	582
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	583 var tx *sql.Tx
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	584 tx, err = mw.JSONConn(req).BeginTx(ctx, nil)
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	585 if err != nil {
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	586 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	587 }
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	588 defer tx.Rollback()
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	589
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	590 // Check if he have such a import job first.
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	591 var dummy bool
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	592 err = tx.QueryRowContext(ctx, selectHasNoRunningImportSQL, id).Scan(&dummy)
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	593 switch {
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	594 case err == sql.ErrNoRows:
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	595 err = mw.JSONError{
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	596 Code: http.StatusNotFound,
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	597 Message: fmt.Sprintf("Cannot find import #%d.", id),
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	598 }
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	599 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	600 case err != nil:
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	601 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	602 }
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	603
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	604 if _, err = tx.ExecContext(ctx, deleteLogsSQL, id); err != nil {
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	605 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	606 }
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	607
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	608 if _, err = tx.ExecContext(ctx, deleteImportSQL, id); err != nil {
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	609 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	610 }
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	611
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	612 if err = tx.Commit(); err != nil {
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	613 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	614 }
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	615
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	616 jr = mw.JSONResult{Code: http.StatusNoContent}
1037 a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	617
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	618 return
a04126989d91 Added endpoint to delete job from import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1031 diff changeset	619 }
1192 3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	620
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	621 func reviewImports(req *http.Request) (mw.JSONResult, error) {
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	622
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	623 rs := mw.JSONInput(req).([]models.Review)
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	624
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	625 type reviewResult struct {
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	626 ID int64 `json:"id"`
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	627 Message string `json:"message,omitempty"`
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	628 Error string `json:"error,omitempty"`
1468 5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	629 }
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	630
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	631 results := make([]reviewResult, len(rs))
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	632
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	633 for i := range results {
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	634 results[i].ID = rs[i].ID
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	635 results[i].Message = fmt.Sprintf("Finalizing import #%d in progress.", rs[i].ID)
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	636 }
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	637
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	638 var wg sync.WaitGroup
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	639 var mu sync.Mutex
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	640
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	641 for i := range rs {
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	642 wg.Add(1)
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	643 go func(idx int) {
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	644 defer wg.Done()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	645 rev := &rs[idx]
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	646 msg, err := decideImport(req, rev.ID, string(rev.State))
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	647 mu.Lock()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	648 if err != nil {
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	649 results[idx].Error = err.Error()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	650 }
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	651 results[idx].Message = msg
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	652 mu.Unlock()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	653 }(i)
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	654 }
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	655
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	656 done := make(chan struct{})
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	657 go func() {
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	658 defer close(done)
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	659 wg.Wait()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	660 }()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	661
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	662 select {
5131 52e3980e3462 review decisions controller: Lowered immediate feedback timeout to 5 secs. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5123 diff changeset	663 case <-time.After(5 * time.Second):
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	664 case <-done:
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	665 }
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	666
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	667 out := make([]reviewResult, len(rs))
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	668 mu.Lock()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	669 copy(out, results)
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	670 mu.Unlock()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	671
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	672 return mw.JSONResult{Result: out}, nil
1468 5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	673 }
5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	674
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	675 func reviewImport(req *http.Request) (jr mw.JSONResult, err error) {
1192 3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	676
3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	677 vars := mux.Vars(req)
3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	678 id, _ := strconv.ParseInt(vars["id"], 10, 64)
3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	679 state := vars["state"]
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	680
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	681 var msg string
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4791 diff changeset	682 if msg, err = decideImport(req, id, state); err != nil {
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	683 return
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	684 }
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	685
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	686 result := struct {
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	687 Message string `json:"message"`
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	688 }{
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	689 Message: msg,
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	690 }
286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	691
4244 4394daeea96a Moved JSONHandler into middleware package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4243 diff changeset	692 jr = mw.JSONResult{Result: &result}
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	693 return
1468 5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	694 }
1192 3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	695
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	696 func decideImport(
1468 5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	697 req *http.Request,
5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	698 id int64,
5e1218b5a123 proof of concept Thomas Junk <thomas.junk@intevation.de> parents: 1392 diff changeset	699 state string,
1469 286a3306f6bf Import bulk review: Cleaned up input JSON document handling. Added proper error handling. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1468 diff changeset	700 ) (message string, err error) {
1192 3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	701
5028 d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4791 diff changeset	702 session, _ := auth.GetSession(req)
d727641911a5 Moved import desision logic to import queue (where it belongs). Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 4791 diff changeset	703 reviewer := session.User
1192 3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	704
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	705 ctx := req.Context()
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	706 accepted := state == "accepted"
eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	707
5153 adf7b9f1273b Send the info if an import job was re-scheduled to sync callers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5131 diff changeset	708 switch err = imports.DecideImport(ctx, id, accepted, reviewer); {
adf7b9f1273b Send the info if an import job was re-scheduled to sync callers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5131 diff changeset	709 case err == imports.ErrRetrying:
adf7b9f1273b Send the info if an import job was re-scheduled to sync callers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5131 diff changeset	710 return "", fmt.Errorf("Finalizing import #%d is re-scheduled.", id)
adf7b9f1273b Send the info if an import job was re-scheduled to sync callers. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5131 diff changeset	711 case err != nil:
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	712 return "", err
1355 b0ebb77f4ab0 Return nice JSON result if import was accepted or declined. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1351 diff changeset	713 }
b0ebb77f4ab0 Return nice JSON result if import was accepted or declined. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1351 diff changeset	714
5123 eeb45e3e0a5a Added mechanism to have sync import jobs on import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 5109 diff changeset	715 return fmt.Sprintf("Import #%d is %s.", id, state), nil
1192 3afa71405b87 Added REST endpoint to accept or decline an import. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1189 diff changeset	716 }

Mercurial > gemma

annotate pkg/controllers/importqueue.go @ 5562:5152b4db40cc