gemma: pkg/imports/sr.go annotate

annotate pkg/imports/sr.go @ 3219:4acbee65275d

Import queue: Split locked dependencies in exclusively and multiple uses.

author	Sascha L. Teichmann <sascha.teichmann@intevation.de>
date	Thu, 09 May 2019 12:49:53 +0200
parents	27ffd94afcb5
children	83b58f6356e7

rev	line source
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	1 // This is Free Software under GNU Affero General Public License v >= 3.0
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	2 // without warranty, see README.md and license for details.
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	3 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	4 // SPDX-License-Identifier: AGPL-3.0-or-later
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	5 // License-Filename: LICENSES/AGPL-3.0.txt
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	6 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	7 // Copyright (C) 2018 by via donau
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	8 // – Österreichische Wasserstraßen-Gesellschaft mbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	9 // Software engineering by Intevation GmbH
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	10 //
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	11 // Author(s):
a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	12 // * Sascha L. Teichmann <sascha.teichmann@intevation.de>
1337 53057ab27ff2 add headers for licensing to some middleware files Fadi Abbud <fadi.abbud@intevation.de> parents: 1292 diff changeset	13 // * Tom Gottfried <tom@intevation.de>
1017 a244b18cb916 Added GNU Affero General Public License. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 991 diff changeset	14
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	15 package imports
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	16
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	17 import (
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	18 "archive/zip"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	19 "bufio"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	20 "context"
973 b6fec8f85599 Generate TINs and octrees in sounding result importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 972 diff changeset	21 "crypto/sha1"
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	22 "database/sql"
973 b6fec8f85599 Generate TINs and octrees in sounding result importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 972 diff changeset	23 "encoding/hex"
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	24 "errors"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	25 "fmt"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	26 "io"
1184 064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	27 "math"
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	28 "os"
960 e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	29 "path"
978 544a5cfe07cd Started with endpoint for uploading sounding result and trigger respective import job. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 977 diff changeset	30 "path/filepath"
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	31 "strconv"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	32 "strings"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	33 "time"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	34
960 e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	35 shp "github.com/jonas-p/go-shp"
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	36
1224 bc4b642c8d04 Started with an upload sounding result to temp upload area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	37 "gemma.intevation.de/gemma/pkg/common"
bc4b642c8d04 Started with an upload sounding result to temp upload area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	38 "gemma.intevation.de/gemma/pkg/models"
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	39 "gemma.intevation.de/gemma/pkg/octree"
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	40 )
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	41
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	42 // SoundingResult is a Job to import sounding reults
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	43 // from a ZIP file into the database.
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	44 type SoundingResult struct {
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	45 // Dir is the folder in the file system where the
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	46 // 'sr.zip' is expect to be.
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	47 Dir string `json:"dir"`
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	48
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	49 // Override data
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	50 // Date if given overrides the date value from the meta.json.
1900 6a67cd819e93 To prepare stretch import made some model data types re-usable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	51 Date *models.Date `json:"date,omitempty"`
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	52 // Date if given overrides the name of the bottleneck from the meta.json.
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	53 Bottleneck *string `json:"bottleneck,omitempty"`
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	54 // EPSG if given overrides the EPSG code from the meta.json.
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	55 // Defaults to WGS84.
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	56 EPSG *uint `json:"epsg,omitempty"`
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	57 // DepthReference if given overides the DepthReference value
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	58 // from the meta.json.
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	59 DepthReference *string `json:"depth-reference,omitempty"`
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	60 }
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	61
964 1e2dce348cfb Serialize boundary polygon of sounding result as WKB. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 961 diff changeset	62 const (
1130 42617bba8709 Go from 50cm to 10cm stepwidth in calculating the contorlines. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1017 diff changeset	63 contourStepWidth = 0.1
977 4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	64 contourTolerance = 0.1
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	65 )
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	66
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	67 // SRJobKind is the unique name of a SoundingResult import job.
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 988 diff changeset	68 const SRJobKind JobKind = "sr"
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 988 diff changeset	69
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1131 diff changeset	70 type srJobCreator struct{}
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1131 diff changeset	71
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	72 func init() {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	73 RegisterJobCreator(SRJobKind, srJobCreator{})
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	74 }
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	75
1754 807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	76 func (srJobCreator) Description() string { return "sounding results" }
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	77
807569b08513 Import queue: Auto acceptance is now a property of the import kind itself and is not configurable any more. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1664 diff changeset	78 func (srJobCreator) AutoAccept() bool { return false }
1646 a0982c38eac0 Import queue: Implemented email notifications. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1572 diff changeset	79
2187 7c83b5277c1c Import queue: Removed boilerplate code to deserialize jobs from JSON by making it part of the import queue. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2168 diff changeset	80 func (srJobCreator) Create() Job { return new(SoundingResult) }
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1131 diff changeset	81
3219 4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2963 diff changeset	82 func (srJobCreator) Depends() [2][]string {
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2963 diff changeset	83 return [2][]string{
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2963 diff changeset	84 {"sounding_results", "sounding_results_contour_lines"},
4acbee65275d Import queue: Split locked dependencies in exclusively and multiple uses. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2963 diff changeset	85 {"bottlenecks"},
1136 a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1131 diff changeset	86 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1131 diff changeset	87 }
a5069da2f0b7 Independent imports in terms of affected tables/dependencies are now run concurrently. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1131 diff changeset	88
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	89 func (srJobCreator) StageDone(
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1317 diff changeset	90 ctx context.Context,
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	91 tx *sql.Tx,
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	92 id int64,
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	93 ) error {
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	94 _, err := tx.ExecContext(ctx, srStageDoneSQL, id)
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	95 return err
991 a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 988 diff changeset	96 }
a301d240905f Decoupled import job creation and job execution with a factory function. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 988 diff changeset	97
977 4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	98 const (
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	99 srStageDoneSQL = `
58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	100 UPDATE waterway.sounding_results SET staging_done = true
1502 0abcc82ef976 Sounding results: Be a bit more precise when setting stageing_done. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1501 diff changeset	101 WHERE id = (
1995 59055c8301df Move import queue to its own database namespace Tom Gottfried <tom@intevation.de> parents: 1900 diff changeset	102 SELECT key from import.track_imports
1502 0abcc82ef976 Sounding results: Be a bit more precise when setting stageing_done. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1501 diff changeset	103 WHERE import_id = $1 AND
0abcc82ef976 Sounding results: Be a bit more precise when setting stageing_done. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1501 diff changeset	104 relation = 'waterway.sounding_results'::regclass)`
1193 58acc343b1b6 Implemented the db stuff of the review process. Needs testing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1184 diff changeset	105
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	106 insertHullSQL = `
961 a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	107 INSERT INTO waterway.sounding_results (
a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	108 bottleneck_id,
a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	109 date_info,
a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	110 depth_reference,
a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	111 area
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	112 ) SELECT
1572 056a86b24be2 Made bottleneck primary key an int. Attention: This may break something! Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1531 diff changeset	113 (SELECT id from waterway.bottlenecks where objnam = $1),
961 a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	114 $2::date,
a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	115 $3,
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	116 (SELECT
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	117 CASE WHEN $5::bytea IS NULL THEN
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	118 ST_Transform(ST_ConcaveHull(ST_Force2D(ST_GeomFromWKB($4, $6::integer)), 0.7), 4326)::geography
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	119 ELSE
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	120 ST_Transform(ST_GeomFromWKB($5, $6::integer), 4326)::geography
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	121 END)
971 f9fb6c399f3f Moved generating of tins to octree package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 964 diff changeset	122 RETURNING
f9fb6c399f3f Moved generating of tins to octree package. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 964 diff changeset	123 id,
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	124 ST_X(ST_Centroid(area::geometry)),
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	125 ST_Y(ST_Centroid(area::geometry)),
2833 1b6840093eac Prevent calculation of wrong UTM zones Tom Gottfried <tom@intevation.de> parents: 2612 diff changeset	126 best_utm(area),
1b6840093eac Prevent calculation of wrong UTM zones Tom Gottfried <tom@intevation.de> parents: 2612 diff changeset	127 ST_AsBinary(ST_Transform(area::geometry, best_utm(area)))
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	128 `
973 b6fec8f85599 Generate TINs and octrees in sounding result importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 972 diff changeset	129
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	130 reprojectPointsSQL = `
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	131 SELECT ST_AsBinary(ST_Transform(ST_GeomFromWKB($1, $2::integer), $3::integer))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	132 `
973 b6fec8f85599 Generate TINs and octrees in sounding result importer. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 972 diff changeset	133 insertOctreeSQL = `
1131 2e6b47cdb2ca Store octrees along with sounding results Tom Gottfried <tom@intevation.de> parents: 1130 diff changeset	134 UPDATE waterway.sounding_results SET
2e6b47cdb2ca Store octrees along with sounding results Tom Gottfried <tom@intevation.de> parents: 1130 diff changeset	135 octree_checksum = $2, octree_index = $3
2e6b47cdb2ca Store octrees along with sounding results Tom Gottfried <tom@intevation.de> parents: 1130 diff changeset	136 WHERE id = $1`
977 4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	137
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	138 insertContourSQL = `
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	139 INSERT INTO waterway.sounding_results_contour_lines (
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	140 sounding_result_id,
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	141 height,
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	142 lines
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	143 )
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	144 SELECT
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	145 $1,
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	146 $2,
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	147 ST_Transform(
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	148 ST_Multi(
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	149 ST_CollectionExtract(
1137 2fb6c0a6ec8a Avoid intersection of each isoline with area on insert Tom Gottfried <tom@intevation.de> parents: 1136 diff changeset	150 ST_SimplifyPreserveTopology(
1410 eb82076fdffe Sounding result import: Ensure that the generated iso-lines are valid. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1392 diff changeset	151 ST_Multi(ST_Collectionextract(
eb82076fdffe Sounding result import: Ensure that the generated iso-lines are valid. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1392 diff changeset	152 ST_MakeValid(ST_GeomFromWKB($4, $3::integer)), 2)),
1137 2fb6c0a6ec8a Avoid intersection of each isoline with area on insert Tom Gottfried <tom@intevation.de> parents: 1136 diff changeset	153 $5
977 4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	154 ),
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	155 2
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	156 )
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	157 ),
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	158 4326
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	159 )
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	160 FROM waterway.sounding_results sr
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	161 WHERE id = $1
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	162 `
961 a4c92e0ef2e1 Stored point cloud and bounding area of sounding result in database. WIP. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 960 diff changeset	163 )
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	164
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	165 // Do executes the actual sounding result import.
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	166 func (sr *SoundingResult) Do(
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1317 diff changeset	167 ctx context.Context,
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	168 importID int64,
930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	169 conn *sql.Conn,
930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	170 feedback Feedback,
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	171 ) (interface{}, error) {
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	172
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	173 begin := time.Now()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	174
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	175 z, err := zip.OpenReader(filepath.Join(sr.Dir, "sr.zip"))
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	176 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	177 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	178 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	179 defer z.Close()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	180
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1137 diff changeset	181 feedback.Info("Looking for 'meta.json'")
1225 4d7c44f7044e Factored out som zip lookup code to be reusable in sounding result upload controller. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1224 diff changeset	182 mf := common.FindInZIP(z, "meta.json")
1292 5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	183 if mf == nil && !sr.completeOverride() {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	184 return nil, errors.New("Cannot find 'meta.json'")
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	185 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	186
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	187 m, err := sr.loadMeta(mf)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	188 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	189 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	190 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	191
2873 b1707f60f241 SR import: Log name of bottleneck and survey date into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2833 diff changeset	192 feedback.Info("Bottleneck: %s", m.Bottleneck)
b1707f60f241 SR import: Log name of bottleneck and survey date into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2833 diff changeset	193 feedback.Info("Survey date: %s", m.Date.Format(common.DateFormat))
b1707f60f241 SR import: Log name of bottleneck and survey date into import log. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2833 diff changeset	194
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1317 diff changeset	195 if err := m.Validate(ctx, conn); err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	196 return nil, common.ToError(err)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	197 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	198
1528 5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	199 var xyzf *zip.File
5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	200 for _, ext := range []string{".xyz", ".txt"} {
5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	201 feedback.Info("Looking for '*%s'", ext)
5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	202 if xyzf = common.FindInZIP(z, ext); xyzf != nil {
5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	203 break
5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	204 }
5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	205 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	206 if xyzf == nil {
1528 5874cedd7f91 Sounding result import: Accept .txt files for XYZ data, too. Sascha L. Teichmann <sascha.teichmann@intevation.de>* parents: 1502 diff changeset	207 return nil, errors.New("Cannot find any .xyz or .txt file")
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	208 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	209
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	210 xyz, err := loadXYZ(xyzf, feedback)
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	211 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	212 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	213 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	214
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	215 if len(xyz) == 0 {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	216 return nil, errors.New("XYZ does not contain any vertices")
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	217 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	218
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	219 // Is there a boundary shapefile in the ZIP archive?
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	220 polygon, err := loadBoundary(z)
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	221 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	222 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	223 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	224
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	225 tx, err := conn.BeginTx(ctx, nil)
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	226 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	227 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	228 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	229 defer tx.Rollback()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	230
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	231 var (
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	232 id int64
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	233 epsg uint32
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	234 lat, lon float64
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	235 )
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	236 start := time.Now()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	237
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	238 var hull []byte
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	239
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	240 xyzWKB := xyz.AsWKB()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	241
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	242 err = tx.QueryRow(insertHullSQL,
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	243 m.Bottleneck,
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	244 m.Date.Time,
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	245 m.DepthReference,
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	246 xyzWKB,
1496 3bf1f0de0763 Un-export point, line string and polygon types in import package as these are only used locally. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1410 diff changeset	247 polygon.asWKB(),
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	248 m.EPSG,
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	249 ).Scan(
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	250 &id,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	251 &lat,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	252 &lon,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	253 &epsg,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	254 &hull,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	255 )
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	256 xyz, polygon = nil, nil // not need from now on.
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	257 feedback.Info("Calculating hull took %s.", time.Since(start))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	258 if err != nil {
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	259 return nil, err
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	260 }
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	261 feedback.Info("Best suited UTM EPSG: %d", epsg)
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	262
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	263 start = time.Now()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	264
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	265 var clippingPolygon octree.Polygon
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	266 if err := clippingPolygon.FromWKB(hull); err != nil {
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	267 return nil, err
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	268 }
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	269 clippingPolygon.Indexify()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	270 feedback.Info("Building clipping polygon took %v.", time.Since(start))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	271
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	272 start = time.Now()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	273
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	274 var reproj []byte
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	275
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	276 if err = tx.QueryRow(reprojectPointsSQL,
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	277 xyzWKB,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	278 m.EPSG,
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	279 epsg,
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	280 ).Scan(&reproj); err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	281 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	282 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	283
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	284 if err := xyz.FromWKB(reproj); err != nil {
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	285 return nil, err
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	286 }
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	287
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	288 feedback.Info("Reprojecting points took %v.", time.Since(start))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	289 feedback.Info("Number of reprojected points: %d", len(xyz))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	290
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	291 start = time.Now()
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	292
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	293 tri, err := octree.Triangulate(xyz)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	294 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	295 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	296 }
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	297 feedback.Info("Triangulation took %v.", time.Since(start))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	298
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	299 start = time.Now()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	300
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	301 tin := tri.Tin()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	302
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	303 var str octree.STRTree
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	304 str.Build(tin)
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	305 feedback.Info("Building STR tree took %v", time.Since(start))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	306
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	307 start = time.Now()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	308
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	309 removed := str.Clip(&clippingPolygon)
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	310 feedback.Info("Clipping STR tree took %v.", time.Since(start))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	311 feedback.Info("Number of triangles to clip %d.", len(removed))
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	312
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	313 start = time.Now()
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	314
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	315 tin.EPSG = epsg
3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	316
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	317 builder := octree.NewBuilder(tin)
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	318 builder.Build(removed)
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	319 octreeIndex, err := builder.Bytes()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	320 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	321 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	322 }
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	323 feedback.Info("Building octree took %v.", time.Since(start))
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	324
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	325 start = time.Now()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	326 h := sha1.New()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	327 h.Write(octreeIndex)
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	328 checksum := hex.EncodeToString(h.Sum(nil))
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	329 _, err = tx.Exec(insertOctreeSQL, id, checksum, octreeIndex)
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	330 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	331 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	332 }
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	333 feedback.Info("Storing octree index took %s.", time.Since(start))
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	334
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	335 tree := builder.Tree()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	336
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	337 start = time.Now()
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	338 err = generateContours(tree, tx, id)
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	339 if err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	340 return nil, err
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	341 }
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	342 feedback.Info("Generating and storing contour lines took %s.",
3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	343 time.Since(start))
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	344
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	345 // Store for potential later removal.
1328 d753ce6cf588 To make golint happier made context.Context to be the first argument in all calls. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1317 diff changeset	346 if err = track(ctx, tx, importID, "waterway.sounding_results", id); err != nil {
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	347 return nil, err
1168 930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	348 }
930fdd8b474f Track successfull imports in a separate table to be able to remove them later. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1138 diff changeset	349
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	350 if err = tx.Commit(); err != nil {
45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	351 feedback.Error(
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	352 "Storing sounding result failed after %v.", time.Since(begin))
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	353 return nil, err
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1137 diff changeset	354 }
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	355
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	356 feedback.Info(
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	357 "Storing sounding result was successful after %v.", time.Since(begin))
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	358
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	359 summary := struct {
1900 6a67cd819e93 To prepare stretch import made some model data types re-usable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	360 Bottleneck string `json:"bottleneck"`
6a67cd819e93 To prepare stretch import made some model data types re-usable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	361 Date models.Date `json:"date"`
6a67cd819e93 To prepare stretch import made some model data types re-usable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	362 Lat float64 `json:"lat"`
6a67cd819e93 To prepare stretch import made some model data types re-usable. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1786 diff changeset	363 Lon float64 `json:"lon"`
1392 0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	364 }{
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	365 Bottleneck: m.Bottleneck,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	366 Date: m.Date,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	367 Lat: lat,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	368 Lon: lon,
0e1d89241cda Imports: An Import (e.g. a sounding result import) can now write a 'summary' of a successful import. This is done if the import switches to to state 'pending'. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1384 diff changeset	369 }
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	370
3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	371 return &summary, nil
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	372 }
7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	373
1498 5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	374 // CleanUp removes the folder containing the ZIP file with the
5df748916fcf Finished API docs for sounding result imports. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1496 diff changeset	375 // the sounding result import.
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	376 func (sr *SoundingResult) CleanUp() error {
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	377 return os.RemoveAll(sr.Dir)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	378 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	379
1292 5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	380 func (sr *SoundingResult) completeOverride() bool {
1384 8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	381 // sr.EPSG == nil -> WGS84
8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	382 return sr.Bottleneck != nil && sr.Date != nil && sr.DepthReference != nil
1292 5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	383 }
5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	384
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	385 func (sr SoundingResult) loadMeta(f zip.File) (*models.SoundingResultMeta, error) {
1292 5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	386 if f == nil {
1384 8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	387 var epsg uint
8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	388 if sr.EPSG != nil {
8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	389 epsg = *sr.EPSG
8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	390 } else {
8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	391 epsg = models.WGS84
8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	392 }
1292 5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	393 return &models.SoundingResultMeta{
5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	394 Date: *sr.Date,
5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	395 Bottleneck: *sr.Bottleneck,
1384 8d59457a5651 Sounding result import: Consider parameter override as complete even if there is no EPSG code. Use WGS84 in this case as a default. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1338 diff changeset	396 EPSG: epsg,
1292 5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	397 DepthReference: *sr.DepthReference,
5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	398 }, nil
5aeda02c51b9 Sounding result import: Don't fail if an import ZIP does not contain an meta.json but has a full set of overrides. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1239 diff changeset	399 }
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	400 r, err := f.Open()
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	401 if err != nil {
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	402 return nil, err
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	403 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	404 defer r.Close()
1224 bc4b642c8d04 Started with an upload sounding result to temp upload area. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1193 diff changeset	405 var m models.SoundingResultMeta
1239 d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	406 if err := m.Decode(r); err != nil {
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	407 return nil, err
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	408 }
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	409
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	410 // Apply overrides
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	411 if sr.Date != nil {
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	412 m.Date = *sr.Date
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	413 }
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	414 if sr.Bottleneck != nil {
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	415 m.Bottleneck = *sr.Bottleneck
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	416 }
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	417 if sr.EPSG != nil {
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	418 m.EPSG = *sr.EPSG
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	419 }
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	420 if sr.DepthReference != nil {
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	421 m.DepthReference = *sr.DepthReference
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	422 }
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	423
d842d9d10872 Sounding result import: Added the feature to override bottleneck, EPSG, depth reference and date in meta.json by POST arguments. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1225 diff changeset	424 return &m, nil
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	425 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	426
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	427 func loadXYZReader(r io.Reader, feedback Feedback) (octree.MultiPointZ, error) {
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	428 mpz := make(octree.MultiPointZ, 0, 250000)
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	429 s := bufio.NewScanner(r)
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	430
2612 c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	431 var hasNegZ bool
c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	432
2963 27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	433 const maxWarnings = 100
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	434 var warnings int
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	435
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	436 warn := func(format string, args ...interface{}) {
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	437 if warnings++; warnings <= maxWarnings {
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	438 feedback.Warn(format, args...)
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	439 }
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	440 }
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	441 defer func() {
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	442 if warnings > maxWarnings {
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	443 feedback.Warn("Too many warnings. %d ignored.", warnings-maxWarnings)
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	444 }
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	445 }()
27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	446
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	447 for line := 1; s.Scan(); line++ {
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	448 text := s.Text()
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	449 var p octree.Vertex
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	450 // fmt.Sscanf(text, "%f,%f,%f") is 4 times slower.
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	451 idx := strings.IndexByte(text, ',')
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	452 if idx == -1 {
2963 27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	453 warn("format error in line %d", line)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	454 continue
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	455 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	456 var err error
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	457 if p.X, err = strconv.ParseFloat(text[:idx], 64); err != nil {
2963 27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	458 warn("format error in line %d: %v", line, err)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	459 continue
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	460 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	461 text = text[idx+1:]
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	462 if idx = strings.IndexByte(text, ','); idx == -1 {
1138 443fc80a315f Don't issue new lines at end of log messages when importing. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1137 diff changeset	463 feedback.Warn("format error in line %d", line)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	464 continue
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	465 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	466 if p.Y, err = strconv.ParseFloat(text[:idx], 64); err != nil {
2963 27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	467 warn("format error in line %d: %v", line, err)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	468 continue
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	469 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	470 text = text[idx+1:]
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	471 if p.Z, err = strconv.ParseFloat(text, 64); err != nil {
2963 27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	472 warn("format error in line %d: %v", line, err)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	473 continue
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	474 }
2612 c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	475 if p.Z < 0 {
c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	476 p.Z = -p.Z
c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	477 if !hasNegZ {
c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	478 hasNegZ = true
2963 27ffd94afcb5 SR import: Limit number of warnings in parsing XYZ files to 100. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2873 diff changeset	479 warn("Negative Z value found: Using -Z")
2612 c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	480 }
c7ce8b011bcb SR import: Negate negative Z values and issue a warning if a negative Z value is found. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2530 diff changeset	481 }
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	482 mpz = append(mpz, p)
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	483 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	484
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	485 if err := s.Err(); err != nil {
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	486 return nil, err
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	487 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	488
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	489 return mpz, nil
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	490 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	491
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	492 func loadXYZ(f *zip.File, feedback Feedback) (octree.MultiPointZ, error) {
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	493 r, err := f.Open()
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	494 if err != nil {
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	495 return nil, err
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	496 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	497 defer r.Close()
988 7dfd3db94e6d In preparation of persisting import jobs logging is done through an interface. Sascha L. Teichmann <teichmann@intevation.de> parents: 984 diff changeset	498 return loadXYZReader(r, feedback)
959 6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	499 }
6ab012d0f0c2 Started with writing an importer job for sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: diff changeset	500
1786 09349ca27dd7 Imports: Removed duplicated code path to store WKB polygons into database. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1754 diff changeset	501 func loadBoundary(z *zip.ReadCloser) (polygonSlice, error) {
1225 4d7c44f7044e Factored out som zip lookup code to be reusable in sounding result upload controller. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1224 diff changeset	502 shpF := common.FindInZIP(z, ".shp")
960 e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	503 if shpF == nil {
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	504 return nil, nil
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	505 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	506 prefix := strings.TrimSuffix(shpF.Name, path.Ext(shpF.Name))
1225 4d7c44f7044e Factored out som zip lookup code to be reusable in sounding result upload controller. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 1224 diff changeset	507 dbfF := common.FindInZIP(z, prefix+".dbf")
960 e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	508 if dbfF == nil {
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	509 return nil, fmt.Errorf("No DBF file found for %s", shpF.Name)
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	510 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	511
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	512 shpR, err := shpF.Open()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	513 if err != nil {
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	514 return nil, err
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	515 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	516
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	517 dbfR, err := dbfF.Open()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	518 if err != nil {
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	519 shpR.Close()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	520 return nil, err
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	521 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	522 sr := shp.SequentialReaderFromExt(shpR, dbfR)
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	523 defer sr.Close()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	524
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	525 if !sr.Next() {
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	526 return nil, sr.Err()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	527 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	528
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	529 _, s := sr.Shape()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	530 if s == nil {
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	531 return nil, sr.Err()
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	532 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	533
972 17a03a84b0e8 Split out polygon code out of sounding result importer source file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 971 diff changeset	534 return shapeToPolygon(s)
960 e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	535 }
e23ae2c83427 Load boundary polygon of sounding result from ZIP file. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 959 diff changeset	536
977 4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	537 func generateContours(tree octree.Tree, tx sql.Tx, id int64) error {
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	538 stmt, err := tx.Prepare(insertContourSQL)
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	539 if err != nil {
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	540 return err
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	541 }
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	542 defer stmt.Close()
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	543
1184 064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	544 // Adjust contour lines heights to multiples of contourStepWidth
2530 3f61b84ae7a6 SR import: Set missing EPSG code to TIN. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2529 diff changeset	545 var heights []float64
1184 064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	546 h := contourStepWidth * math.Ceil(tree.Min.Z/contourStepWidth)
064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	547 for ; h <= tree.Max.Z; h += contourStepWidth {
064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	548 heights = append(heights, h)
064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	549 }
064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	550
064d44ccc6f2 Adjust contour lines heights to multiples of step width Tom Gottfried <tom@intevation.de> parents: 1168 diff changeset	551 octree.DoContours(tree, heights, func(res *octree.ContourResult) {
2529 45d51a49f191 SR import: Use own triangulation and clipping when importing sounding results. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 2516 diff changeset	552 if err == nil && len(res.Lines) > 0 {
977 4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	553 _, err = stmt.Exec(
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	554 id, res.Height, tree.EPSG,
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	555 res.Lines.AsWKB2D(),
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	556 contourTolerance)
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	557 }
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	558 })
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	559
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	560 return err
4a2ca0e20006 Fixed build error. Sascha L. Teichmann <sascha.teichmann@intevation.de> parents: 976 diff changeset	561 }

Mercurial > gemma

annotate pkg/imports/sr.go @ 3219:4acbee65275d