Updated with v5.0.0

CHANGELOG.md

@@ -1,3 +1,11 @@
+## 0.5.0 (2017-12-XX)
+- Added: Command line `cache seed` and `cache purge` commands
+- Added: More robust command line interface
+- Fixed: Possible Panic if a feature without an ID is added before a feature with an ID; when constructing Layers (#195)
+
+Breaking changes:
+- To use tegola as a web server, use the command `tegola serve --config=/path/to/config.toml`
+
 ## 0.4.2 (2017-11-28)
 - Fixed: Performance affected by unused log statements (#197, @remster)
 
@@ -70,4 +78,4 @@
 - Added: concurrent layer fetching from data providers.
 - Added: remote config loading over http(s).
 
-## v0.1.0 (2016-07-29)
+## v0.1.0 (2016-07-29)

README.md

@@ -9,19 +9,41 @@ Tegola is a vector tile server delivering [Mapbox Vector Tiles](https://github.c
 
 ## Features
 - Native geometry processing (simplification, clipping, make valid, intersection, contains, scaling, translation)
-- [Mapbox Vector Tile v2 specification](https://github.com/mapbox/vector-tile-spec) compliant .
+- [Mapbox Vector Tile v2 specification](https://github.com/mapbox/vector-tile-spec) compliant.
 - Embedded viewer with auto generated style for quick data visualization and inspection.
 - Support for PostGIS as a data provider. Extensible to support additional data providers.
 - Local filesystem caching. Extensible design to support additional cache backends.
+- Cache seeding to fill the cache prior to web requests.
 - Parallelized tile serving and geometry processing.
 - Support for Web Mercator (3857) and WGS84 (4326) projections.
 
-## Running Tegola
+## Usage
+```
+tegola is a vector tile server
+Version: v0.5.0
+
+Usage:
+  tegola [command]
+
+Available Commands:
+  cache       Manipulate the tile cache
+  help        Help about any command
+  serve       Use tegola as a tile server
+  version     Print the version number of tegola
+
+Flags:
+      --config string   path to config file (default "config.toml")
+  -h, --help            help for tegola
+
+Use "tegola [command] --help" for more information about a command.
+```
+
+## Running tegola as a vector tile server
 1. Download the appropriate binary of tegola for your platform via the [release page](https://github.com/terranodo/tegola/releases).
 2. Setup your config file and run. Tegola expects a `config.toml` to be in the same directory as the binary. You can set a different location for the `config.toml` using a command flag:
 
 ```
-./tegola -config=/path/to/config.toml
+./tegola sever --config=/path/to/config.toml
 ```
 
 ## Server Endpoints
@@ -74,14 +96,12 @@ Return [TileJSON](https://github.com/mapbox/tilejson-spec) details about the map
 
 Return an auto generated [Mapbox GL Style](https://www.mapbox.com/mapbox-gl-js/style-spec/) for the configured map.
 
-
 ## Configuration
 The tegola config file uses the [TOML](https://github.com/toml-lang/toml) format. The following example shows how to configure a PostGIS data provider with two layers. The first layer includes a `tablename`, `geometry_field` and an `id_field`. The second layer uses a custom `sql` statement instead of the `tablename` property.
 
 Under the `maps` section, map layers are associated with data provider layers and their `min_zoom` and `max_zoom` values are defined. Optionally, `default_tags` can be setup which will be encoded into the layer. If the same tags are returned from a data provider, the data provider's values will take precedence.
 
 ```toml
-
 [webserver]
 port = ":9090"              # port to bind the web server to. defaults ":8080"
 
@@ -116,9 +136,9 @@ max_connections = "50"      # The max connections to maintain in the connection
 	fields = [ "class", "name" ]        # Additional fields to include in the select statement.
 
 	[[providers.layers]]
-	name = "rivers" 					# will be encoded as the layer name in the tile
+	name = "rivers"                     # will be encoded as the layer name in the tile
 	geometry_fieldname = "geom"         # geom field. default is geom
-	id_fieldname = "gid"              # geom id field. default is gid
+	id_fieldname = "gid"                # geom id field. default is gid
 	# Custom sql to be used for this layer. Note: that the geometery field is wraped
 	# in a ST_AsBinary, as tegola only understand wkb.
 	sql = """
@@ -140,7 +160,7 @@ name = "zoning"                              # used in the URL to reference this
 	min_zoom = 12                            # minimum zoom level to include this layer
 	max_zoom = 16                            # maximum zoom level to include this layer
 
-		[maps.layers.default_tags]		     # table of default tags to encode in the tile. SQL statements will override
+		[maps.layers.default_tags]           # table of default tags to encode in the tile. SQL statements will override
 		class = "park"
 
 	[[maps.layers]]
@@ -155,43 +175,29 @@ The following tokens are supported in custom SQL queries for the PostGIS data pr
 - `!BBOX!` - [required] Will convert the z/x/y values into a bounding box to query the feature table with.
 - `!ZOOM!` - [optional] Pass in the zoom value for the request. Useful for filtering feature results by zoom.
 
-## Command line flags
-Tegola supports the following command flags:
-
-- `config` - Location of config file in TOML format. Can be absolute, relative or remote over http(s).
-- `port` - Port for the webserver to bind to. i.e. :8080
-- `log-file` - Path to write webserver tile request logs
-- `log-format` - The format that the logger will log with. Available fields:
-  - `{{.Time}}` : The current Date Time in RFC 2822 format.
-  - `{{.RequestIP}}` : The IP address of the the requester.
-  - `{{.Z}}` : The Zoom level.
-  - `{{.X}}` : The X Coordinate.
-  - `{{.Y}}` : The Y Coordinate.
-
-## Debugging
-
-### Environment Variables
+## Environment Variables
 The following environment variables can be used for debugging:
 
 `SQL_DEBUG` specify the type of SQL debug information to output. Currently support two values:
 
 - `LAYER_SQL`: print layer SQL as they are parsed from the config file.
 - `EXECUTE_SQL`: print SQL that is executed for each tile request and the number of items it returns or an error.
 
+#### Usage
+
+```bash
+$ SQL_DEBUG=LAYER_SQL tegola -config=/path/to/conf.toml
+```
+
+The following environment variables can be use to control various runtime options:
 
 `TEGOLA_OPTIONS` specify a set of options comma (or space) seperated options.
 
 - `DontSimplifyGeo` to turn off simplification for all layers.
 - `SimplifyMaxZoom={{int}}` to set the max zoom that simplification will apply to. (14 is default)
 
 
-#### Usage
-
-```bash
-$ SQL_DEBUG=LAYER_SQL tegola -config=/path/to/conf.toml
-```
-
-### Client side
+## Client side debugging
 When debugging client side, it's often helpful to to see an outline of a tile along with it's Z/X/Y values. To encode a debug layer into every tile add the query string variable `debug=true` to the URL template being used to request tiles. For example:
 
 ```

atlas/atlas.go

@@ -6,6 +6,8 @@ import (
 
 	"github.com/terranodo/tegola"
 	"github.com/terranodo/tegola/cache"
+	_ "github.com/terranodo/tegola/cache/filecache"
+	_ "github.com/terranodo/tegola/cache/s3cache"
 )
 
 //	DefaultAtlas is instanitated for convenience

cache/errors.go

@@ -20,3 +20,30 @@ type ErrInvalidFileKey struct {
 func (e ErrInvalidFileKey) Error() string {
 	return fmt.Sprintf("cache: invalid fileKey (%v). unable to parse (%v) value (%v) into int", e.path, e.key, e.val)
 }
+
+type ErrGettingFromCache struct {
+	Err       error
+	CacheType string
+}
+
+func (e ErrGettingFromCache) Error() string {
+	return fmt.Sprintf("cache: error getting from (%v) cache: %v", e.CacheType, e.Err)
+}
+
+type ErrSettingToCache struct {
+	Err       error
+	CacheType string
+}
+
+func (e ErrSettingToCache) Error() string {
+	return fmt.Sprintf("cache: error setting to (%v) cache: %v", e.CacheType, e.Err)
+}
+
+type ErrPurgingCache struct {
+	Err       error
+	CacheType string
+}
+
+func (e ErrPurgingCache) Error() string {
+	return fmt.Sprintf("cache: error purging (%v) cache: %v", e.CacheType, e.Err)
+}

cache/filecache/filecache.go

@@ -3,18 +3,15 @@ package filecache
 import (
 	"errors"
 	"io/ioutil"
-	"log"
 	"os"
 	"path/filepath"
-	"sync"
 
 	"github.com/terranodo/tegola/cache"
 	"github.com/terranodo/tegola/util/dict"
 )
 
 var (
 	ErrMissingBasepath = errors.New("filecache: missing required param 'basepath'")
-	ErrCacheMiss       = errors.New("filecache: cache miss")
 )
 
 const CacheType = "file"
@@ -37,9 +34,7 @@ func New(config map[string]interface{}) (cache.Interface, error) {
 	var err error
 
 	//	new filecache
-	fc := Filecache{
-		Locker: map[string]sync.RWMutex{},
-	}
+	fc := Filecache{}
 
 	//	parse the config
 	c := dict.M(config)
@@ -69,54 +64,11 @@ func New(config map[string]interface{}) (cache.Interface, error) {
 		return nil, err
 	}
 
-	//	walk our basepath and fill the filecache Locker with keys for already cached tiles
-	err = filepath.Walk(fc.Basepath, func(path string, info os.FileInfo, err error) error {
-		if err != nil {
-			return err
-		}
-
-		//	skip directories
-		if info.IsDir() {
-			return nil
-		}
-
-		//	remove the basepath for the file key
-		fileKey := path[len(fc.Basepath):]
-
-		cacheKey, err := cache.ParseKey(fileKey)
-		if err != nil {
-			log.Println("filecache: ", err.Error())
-			return nil
-		}
-
-		//	write our key
-		fc.Lock()
-		fc.Locker[cacheKey.String()] = sync.RWMutex{}
-		fc.Unlock()
-
-		return nil
-	})
-	if err != nil {
-		return nil, err
-	}
-
 	return &fc, nil
 }
 
 type Filecache struct {
 	Basepath string
-
-	//	we need a cache mutex to avoid concurrent writes to our Locker
-	sync.RWMutex
-
-	//	Locker tracks which cache keys are being operated on.
-	//	when the cache is being written to a Lock() is set.
-	//	when being read from an RLock() is used so we don't
-	//	block concurrent reads.
-	//
-	//	TODO: store a hash of the cache blob along with the Locker mutex
-	Locker map[string]sync.RWMutex
-
 	//	MaxZoom determins the max zoom the cache to persist. Beyond this
 	//	zoom, cache Set() calls will be ignored. This is useful if the cache
 	//	should not be leveraged for higher zooms when data changes often.
@@ -129,27 +81,8 @@ type Filecache struct {
 func (fc *Filecache) Get(key *cache.Key) ([]byte, bool, error) {
 	path := filepath.Join(fc.Basepath, key.String())
 
-	//	lookup our mutex
-	fc.RLock()
-	mutex, ok := fc.Locker[key.String()]
-	fc.RUnlock()
-	if !ok {
-		//	no entry, return a miss
-		return nil, false, nil
-	}
-
-	//	read lock
-	mutex.RLock()
-	defer mutex.RUnlock()
-
 	f, err := os.Open(path)
 	if err != nil {
-		//	something is wrong with opening this file
-		//	remove the key from the cache if it exists
-		fc.Lock()
-		delete(fc.Locker, key.String())
-		fc.Unlock()
-
 		if os.IsNotExist(err) {
 			return nil, false, nil
 		}
@@ -173,29 +106,20 @@ func (fc *Filecache) Set(key *cache.Key, val []byte) error {
 		return nil
 	}
 
-	path := filepath.Join(fc.Basepath, key.String())
+	//	the tmpPath uses the destPath with a simple "-tmp" suffix. we're going to do
+	//	a Rename at the end of this method and according to the os.Rename() docs:
+	//	"If newpath already exists and is not a directory, Rename replaces it.
+	//	OS-specific restrictions may apply when oldpath and newpath are in different directories"
+	destPath := filepath.Join(fc.Basepath, key.String())
+	tmpPath := destPath + "-tmp"
 
-	//	lookup our mutex
-	fc.RLock()
-	mutex, ok := fc.Locker[key.String()]
-	fc.RUnlock()
-	if !ok {
-		fc.Lock()
-		fc.Locker[key.String()] = sync.RWMutex{}
-		mutex = fc.Locker[key.String()]
-		fc.Unlock()
-	}
 	//	the key can have a directory syntax so we need to makeAll
-	if err = os.MkdirAll(filepath.Dir(path), os.ModePerm); err != nil {
+	if err = os.MkdirAll(filepath.Dir(destPath), os.ModePerm); err != nil {
 		return err
 	}
 
-	//	write lock
-	mutex.Lock()
-	defer mutex.Unlock()
-
 	//	create the file
-	f, err := os.Create(path)
+	f, err := os.Create(tmpPath)
 	if err != nil {
 		return err
 	}
@@ -207,7 +131,8 @@ func (fc *Filecache) Set(key *cache.Key, val []byte) error {
 		return err
 	}
 
-	return nil
+	//	move the temp file to the destination
+	return os.Rename(tmpPath, destPath)
 }
 
 func (fc *Filecache) Purge(key *cache.Key) error {
@@ -219,9 +144,5 @@ func (fc *Filecache) Purge(key *cache.Key) error {
 	}
 
 	//	remove the locker key on purge
-	fc.Lock()
-	delete(fc.Locker, key.String())
-	fc.Unlock()
-
 	return os.Remove(path)
 }