Skip to content

Commit

Permalink
Added settings to set the chunk size and amount of parallel uploads f…
Browse files Browse the repository at this point in the history 
…or faster upload speed, fixes #162. Updated documentation
  • Loading branch information
@Forceu
Forceu committed Jun 7, 2024
1 parent ca63110 commit 3b7cdae
Show file tree
Hide file tree
Showing 11 changed files with 163 additions and 61 deletions.
4 changes: 2 additions & 2 deletions build/go-generate/updateVersionNumbers.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import (
"strings"
)

const versionJsAdmin = "2"
const versionJsDropzone = "3"
const versionJsAdmin = "3"
const versionJsDropzone = "4"
const versionJsE2EAdmin = "2"
const versionCssMain = "2"

Expand Down
124 changes: 93 additions & 31 deletions docs/advanced.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,37 +57,41 @@ Available environment variables
==================================


+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| Name | Action | Persistent [*]_ | Default |
+===========================+=====================================================================================+=================+=============================+
| GOKAPI_CONFIG_DIR | Sets the directory for the config file | No | config |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_CONFIG_FILE | Sets the name of the config file | No | config.json |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_DATA_DIR | Sets the directory for the data | Yes | data |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_DB_NAME | Sets the name for the database file | No | gokapi.sqlite |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_LENGTH_ID | Sets the length of the download IDs. Value needs to be 5 or more | Yes | 15 |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_MAX_FILESIZE | Sets the maximum allowed file size in MB | Yes | 102400 (100GB) |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_MAX_MEMORY_UPLOAD | Sets the amount of RAM in MB that can be allocated for an upload. | Yes | 20 |
| | | | |
| | Any upload with a size greater than that will be written to a temporary file | | |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_PORT | Sets the webserver port | Yes | 53842 |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_DISABLE_CORS_CHECK | Disables the CORS check on startup and during setup, if set to "true" | No | false |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_LOG_STDOUT | Also outputs all log file entries to the console output | No | false |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| DOCKER_NONROOT | Docker only: Runs the binary in the container as a non-root user, if set to "true" | No | false |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| TMPDIR | Sets the path which contains temporary files | No | Non-Docker: Default OS path |
| | | | |
| | | | Docker: [DATA_DIR] |
+---------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| Name | Action | Persistent [*]_ | Default |
+=============================+=====================================================================================+=================+=============================+
| GOKAPI_CHUNK_SIZE_MB | Sets the size of chunks that are uploaded in MB | Yes | 45 |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_CONFIG_DIR | Sets the directory for the config file | No | config |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_CONFIG_FILE | Sets the name of the config file | No | config.json |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_DATA_DIR | Sets the directory for the data | Yes | data |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_DB_NAME | Sets the name for the database file | No | gokapi.sqlite |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_LENGTH_ID | Sets the length of the download IDs. Value needs to be 5 or more | Yes | 15 |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_MAX_FILESIZE | Sets the maximum allowed file size in MB | Yes | 102400 (100GB) |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_MAX_MEMORY_UPLOAD | Sets the amount of RAM in MB that can be allocated for an upload chunk or file | Yes | 50 |
| | | | |
| | Any chunk or file with a size greater than that will be written to a temporary file | | |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_MAX_PARALLEL_UPLOADS | Set the amount of chunks that are uploaded in parallel for a single file | Yes | 4 |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_PORT | Sets the webserver port | Yes | 53842 |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_DISABLE_CORS_CHECK | Disables the CORS check on startup and during setup, if set to “true” | No | false |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| GOKAPI_LOG_STDOUT | Also outputs all log file entries to the console output | No | false |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| DOCKER_NONROOT | Docker only: Runs the binary in the container as a non-root user, if set to “true” | No | false |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+
| TMPDIR | Sets the path which contains temporary files | No | Non-Docker: Default OS path |
| | | | |
| | | | Docker: [DATA_DIR] |
+-----------------------------+-------------------------------------------------------------------------------------+-----------------+-----------------------------+


.. [*] Variables that are persistent must be submitted during the first start when Gokapi creates a new config file. They can be omitted afterwards. Non-persistent variables need to be set on every start.
Expand Down Expand Up @@ -154,6 +158,64 @@ Example: Deleting a file



.. _chunksizes:

*****************************************************************************
Chunk Sizes / Considerations for servers with limited or high amount of RAM
*****************************************************************************

By default, Gokapi uploads files in 45MB chunks stored in RAM. Up to 4 chunks are sent in parallel to enhance upload speed, requiring up to 200MB of RAM per file during upload in the standard configuration.

Servers with limited RAM
================================

To conserve RAM, you can either

* configure Gokapi to save the chunks on disk instead of RAM, by setting the ``MaxMemory`` setting to a value lower than your chunk size
* reduce the chunk size by setting the ``ChunkSize`` to a lower value
* decrease the amount of parallel uploads by setting ``MaxParallelUploads`` to a lower value

Refer to :ref:`chunk_config` for instructions on changing these values.

Servers with high amount of RAM
================================

If your server has a lot of available RAM, you can improve upload speed by increasing the chunk size, which reduces overhead during upload.

* Increase the chunk size by setting the ``ChunkSize`` to a larger value
* Make sure that the ``MaxMemory`` setting is a higher value than your chunk size
* Consider increasing the amount of parallel uploads by setting ``MaxParallelUploads`` to a higher value


Refer to :ref:`chunk_config` for instructions on changing these values.

.. note::
Ensure your reverse proxy and CDN (if applicable) support the chosen chunk size. Cloudflare users on the free tier are limited to 100MB file chunks.


.. _chunk_config:


Changing the configuration
============================

If you have not completed the Gokapi setup yet, you can set all the values mentioned above using environment variables. See :ref:`passingenv` for instructions. If the setup is complete, Gokapi will ignore these environment variables, and you'll need to modify the configuration file (by default: ``config.json`` in the folder ``config``). See the table below on how to change the values:


+----------------------------------------+-----------------------------+--------------------------+---------+
| Configuration | Environment Variable | Configuration File Entry | Default |
+========================================+=============================+==========================+=========+
| Chunk size for uploads | GOKAPI_CHUNK_SIZE_MB | ChunkSize | 45 |
+----------------------------------------+-----------------------------+--------------------------+---------+
| Maximum size for chunks or whole files | GOKAPI_MAX_MEMORY_UPLOAD | MaxMemory | 50 |
| | | | |
| to store in RAM during upload | | | |
+----------------------------------------+-----------------------------+--------------------------+---------+
| Parallel uploads per file | GOKAPI_MAX_PARALLEL_UPLOADS | MaxParallelUploads | 4 |
+----------------------------------------+-----------------------------+--------------------------+---------+




********************************
Automatic Deployment
Expand Down
22 changes: 22 additions & 0 deletions docs/changelog.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,28 @@ Changelog
Overview of all Changes
-----------------------

v1.8.4: 29 May 2024
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Gokapi runs as root in Docker container by default (this was changed in 1.8.3). To run it as unprivileged user, set environment variable DOCKER_NONROOT to true.
* Removed logging of errors when a user unexpectedly closed download or upload connection


v1.8.3: 27 May 2024
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Fixed Keycloak documentation. **Important:** If you have used the old example for configuration, please make sure that it is configure properly, as with the old example unauthorised access might have been possible! Documentation: Creating scopes for groups
* The binary will no longer be run as root in the Docker image. Breaking change: If you want to reconfigure Gokapi, the argument to pass to Docker is now a different one: Documentation
* If salts are empty, new ones will now be generated on startup. This is to aid automatic deployment
* A new admin password can be set with --deployment-password newPassword, but this should only be used for automatic deployment
* Env variable GOKAPI_LOG_STDOUT added, which also outputs all log entries to the terminal
* Display error message, if a reverse proxy does not allow file upload, or has been set to accept a too low file size
* Added header, so that nginx does not cache SSE
* Cloud storage file downloads can now be proxied through Gokapi, e.g. if the storage server is on an internal network
* Fixed a bug, where the option "Always save images locally" reverted back to default when re-running setup
* Updated documentation


v1.8.2: 20 Apr 2024
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
12 changes: 10 additions & 2 deletions internal/configuration/configupgrade/Upgrade.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ import (
)

// CurrentConfigVersion is the version of the configuration structure. Used for upgrading
const CurrentConfigVersion = 18
const CurrentConfigVersion = 19

// DoUpgrade checks if an old version is present and updates it to the current version if required
func DoUpgrade(settings *models.Configuration, env *environment.Environment) bool {
if settings.ConfigVersion < CurrentConfigVersion {
updateConfig(settings, env)
fmt.Println("Successfully upgraded database")
fmt.Println("Successfully upgraded configuration")
settings.ConfigVersion = CurrentConfigVersion
return true
}
Expand Down Expand Up @@ -65,6 +65,14 @@ func updateConfig(settings *models.Configuration, env *environment.Environment)
}
settings.Authentication.OAuthRecheckInterval = 168
}
// < v1.8.5
if settings.ConfigVersion < 19 {
if settings.MaxMemory == 40 {
settings.MaxMemory = 50
}
settings.ChunkSize = env.ChunkSizeMB
settings.MaxParallelUploads = env.MaxParallelUploads
}
}

// migrateToSqlite copies the content of the old bitcask database to a new sqlite database
Expand Down
36 changes: 19 additions & 17 deletions internal/environment/Environment.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,25 @@ const DefaultPort = 53842

// Environment is a struct containing available env variables
type Environment struct {
ConfigDir string `env:"CONFIG_DIR" envDefault:"config"`
ConfigFile string `env:"CONFIG_FILE" envDefault:"config.json"`
ConfigPath string
DataDir string `env:"DATA_DIR" envDefault:"data"`
WebserverPort int `env:"PORT" envDefault:"53842"`
LengthId int `env:"LENGTH_ID" envDefault:"15"`
MaxMemory int `env:"MAX_MEMORY_UPLOAD" envDefault:"40"`
MaxFileSize int `env:"MAX_FILESIZE" envDefault:"102400"` // 102400==100GB
AwsBucket string `env:"AWS_BUCKET"`
AwsRegion string `env:"AWS_REGION"`
AwsKeyId string `env:"AWS_KEY"`
AwsKeySecret string `env:"AWS_KEY_SECRET"`
AwsEndpoint string `env:"AWS_ENDPOINT"`
AwsProxyDownload bool `env:"AWS_PROXY_DOWNLOAD" envDefault:"false"`
DatabaseName string `env:"DB_NAME" envDefault:"gokapi.sqlite"`
DisableCorsCheck bool `env:"DISABLE_CORS_CHECK" envDefault:"false"`
LogToStdout bool `env:"LOG_STDOUT" envDefault:"false"`
ConfigDir string `env:"CONFIG_DIR" envDefault:"config"`
ConfigFile string `env:"CONFIG_FILE" envDefault:"config.json"`
ConfigPath string
DataDir string `env:"DATA_DIR" envDefault:"data"`
WebserverPort int `env:"PORT" envDefault:"53842"`
LengthId int `env:"LENGTH_ID" envDefault:"15"`
MaxMemory int `env:"MAX_MEMORY_UPLOAD" envDefault:"50"`
MaxFileSize int `env:"MAX_FILESIZE" envDefault:"102400"` // 102400==100GB
MaxParallelUploads int `env:"MAX_PARALLEL_UPLOADS" envDefault:"4"`
ChunkSizeMB int `env:"CHUNK_SIZE_MB" envDefault:"45"`
AwsBucket string `env:"AWS_BUCKET"`
AwsRegion string `env:"AWS_REGION"`
AwsKeyId string `env:"AWS_KEY"`
AwsKeySecret string `env:"AWS_KEY_SECRET"`
AwsEndpoint string `env:"AWS_ENDPOINT"`
AwsProxyDownload bool `env:"AWS_PROXY_DOWNLOAD" envDefault:"false"`
DatabaseName string `env:"DB_NAME" envDefault:"gokapi.sqlite"`
DisableCorsCheck bool `env:"DISABLE_CORS_CHECK" envDefault:"false"`
LogToStdout bool `env:"LOG_STDOUT" envDefault:"false"`
// Deprecated: will be removed with 1.9
LegacyDbPath string
// Deprecated: will be removed with 1.9
Expand Down
Loading

0 comments on commit 3b7cdae

Please sign in to comment.