From f42fee5f5042874f1717caec35fc82591e4802e4 Mon Sep 17 00:00:00 2001 From: Zach Musgrave Date: Wed, 28 Dec 2022 18:12:02 -0800 Subject: [PATCH] Rewrote readme and related docs --- ARCHITECTURE.md | 25 +-- BACKEND.md | 135 +++++++++++ CONTRIBUTING.md | 51 ++--- MAINTAINERS | 3 +- README.md | 575 +++++++---------------------------------------- SUPPORTED.md | 201 ----------------- _example/main.go | 3 +- 7 files changed, 234 insertions(+), 759 deletions(-) create mode 100644 BACKEND.md delete mode 100644 SUPPORTED.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 660b7baa3e..8728960e75 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -33,8 +33,9 @@ several main roles: `Expression`, ... - Provides implementations of components used in the rest of the packages `Row`, `Context`, `ProcessList`, `Catalog`, ... -- Defines the `information_schema` table, which is a special database - and contains some information about the schemas of other tables. +- Defines the `information_schema` database, which is a special + database and contains some information about the schemas of other + tables. ### `sql/analyzer` @@ -63,9 +64,6 @@ logic operators, conversions, etc are implemented here. Inside `registry.go` there is a registry of all the default functions, even if they're not defined here. -`Inspect` and `Walk` utility functions are provided to inspect -expressions. - ### `sql/expression/function` Implementation of all the functions available in go-mysql-server. @@ -127,23 +125,6 @@ Contains a function to `Find` the most similar name from an array to a given one using the Levenshtein distance algorithm. Used for suggestions on errors. -## `internal/regex` - -go-mysql-server has multiple regular expression engines, such as -oniguruma and the standard Go regexp engine. In this package, a common -interface for regular expression engines is defined. This means, Go -standard library `regexp` package should not be used in any -user-facing feature, instead this package should be used. - -The default engine is oniguruma, but the Go standard library engine -can be used using the `mysql_go_regex` build tag. - -## `test` - -Test contains pieces that are only used for tests, such as an -opentracing tracer that stores spans in memory to be inspected later -in the tests. - ## `_integration` To ensure compatibility with some clients, there is a small example diff --git a/BACKEND.md b/BACKEND.md new file mode 100644 index 0000000000..5158ef0ba0 --- /dev/null +++ b/BACKEND.md @@ -0,0 +1,135 @@ +# Custom backend integration guide + +This is the guide for creating a new backend to query with +**go-mysql-server**. + +## Core interfaces + +To create your own data source implementation you need to implement +the following interfaces: + +- `sql.DatabaseProvider`. This interface allows the engine to find + available databases. You can also unlock addtional functionality by + optionally implementing additional interfaces: + - `sql.MutableDatabaseProvider` to support creating and dropping + databases. + - `sql.CollatedDatabaseProvider` to support database-level + collations. +- `sql.Database`. These are returned by your + `sql.DatabaseProvider`. The main job of `sql.Database` is to provide + tables from your data source. You can also implement other + interfaces on your database to unlock additional functionality: + - `sql.TableCreator` to support creating new tables + - `sql.TableDropper` to support dropping tables + - `sql.TableRenamer` to support renaming tables + - `sql.ViewCreator` to support creating persisted views on your tables + - `sql.ViewDropper` to support dropping persisted views +- `sql.Table`. This interface will provide rows of values from your + data source. You can also implement other interfaces on your table + to unlock additional functionality: + - `sql.InsertableTable` to allow your data source to be updated with + `INSERT` statements. + - `sql.UpdateableTable` to allow your data source to be updated with + `UPDATE` statements. + - `sql.DeletableTable` to allow your data source to be updated with + `DELETE` statements. + - `sql.ReplaceableTable` to allow your data source to be updated with + `REPLACE` statements. + - `sql.AlterableTable` to allow your data source to have its schema + modified by adding, dropping, and altering columns. + - `sql.IndexedTable` to declare your table's native indexes to speed + up query execution. + - `sql.IndexAlterableTable` to accept the creation of new native + indexes. + - `sql.ForeignKeyAlterableTable` to signal your support of foreign + key constraints in your table's schema and data. + - `sql.ProjectedTable` to return rows that only contain a subset of + the columns in the table. This can make query execution faster. + - `sql.FilteredTable` to filter the rows returned by your table to + those matching a given expression. This can make query execution + faster (if your table implementation can filter rows more + efficiently than checking an expression on every row in a table). + +This is not a complete list, but should be enough to get you started +on a full backend implementation. For an example of implementing these +interfaces, see the `memory` package. + +## Sessions and transactions + +Many backend implementations will be able to re-use the +`sql.BaseSession` object for sessioned access to databases. This +should be the case for all read-only database implementations. +However, some backends may need to store session information +particular to that backend, e.g. open data files that have yet to be +written. Such integrators should implement their own `sql.Session` +implementation, and probably should embed `sql.BaseSession` in it to +make that easier. + +For backends that want transactional semantics for their queries must +also implement `sql.TransactionSession` in their session object and +provide a corresponding `sql.Transaction` implementation. The details +of doing so are necessarily very specific to a particuarl backend and +are beyond the scope of this guide. + +## Native indexes + +Tables can declare that they support native indexes. The `memory` +package contains an example of this behavior, but please note that it +is only for example purposes and doesn't actually make queries faster +(although we could change this in the future). + +Integrators should implement the `sql.IndexedTable` interface to +declare which indexes their tables support and provide a means of +returning a subset of the rows. The job of your `sql.Index` +implementation is to accept or reject combinations of `sql.Range` +expressions that it can support, which will be used by the engine to +construct a `sql.IndexLookup` struct to provide to your `sql.Indexed` +table implementation. + +## Custom index driver implementation + +Index drivers are separate backends for storing and querying indexes, +without the need for a table to store and query its own native +indexes. To implement a custom index driver you need to implement a +few things: + +- `sql.IndexDriver` interface, which will be the driver itself. Not + that your driver must return an unique ID in the `ID` method. This + ID is unique for your driver and should not clash with any other + registered driver. It's the driver's responsibility to be fault + tolerant and be able to automatically detect and recover from + corruption in indexes. +- `sql.Index` interface, returned by your driver when an index is + loaded or created. +- `sql.IndexValueIter` interface, which will be returned by your + `sql.IndexLookup` and should return the values of the index. +- Don't forget to register the index driver in your `sql.Context` + using `context.RegisterIndexDriver(mydriver)` to be able to use it. + +To create indexes using your custom index driver you need to use +extension syntax `USING driverid` on the index creation statement. For +example: + +```sql +CREATE INDEX foo ON table USING driverid (col1, col2) +``` + +**go-mysql-server** does not provide a production index driver +implementation. We previously provided a pilosa implementation, but +removed it due to the difficulty of supporting it on all platforms +(pilosa doesn't work on Windows). + +You can see an example of a driver implementation in the memory +package. + +## Testing your backend implementation + +**go-mysql-server** provides a suite of engine tests that you can use +to validate that your implementation works as expected. See the +`enginetest` package for details and examples. + +It's also possible and encouraged to write engine tests that are +specific to your backend. This is especially important when +implementing transactions, which the in-memory backend doesn't +support. + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c3dba77388..88012409cb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,45 +5,26 @@ This document outlines some of the conventions on development workflow, commit message formatting, contact points, and other resources to make it easier to get your contribution accepted. -## Contributor License Agreement (CLA) - -By contributing to this project you agree to the Contributor License -Agreement (CLA). This grants copyright of your work on the project to -the project owners, Dolthub Inc, as well as waives certain other -rights related to your contribution. The first time you submit a PR, -you will be prompted to read and sign the CLA. We cannot accept -contributions that do not sign the CLA. - ## Support Channel The official support channel, for both users and contributors, is -GitHub issues. +GitHub issues. You can also talk to engineers on the [Dolt Discord +server](https://discord.com/invite/RFwfYpu). ## How to Contribute Pull Requests (PRs) are the exclusive way to contribute code to -go-mysql-server. In order for a PR to be accepted it needs to pass a -list of requirements: - -- The contribution must be correctly explained with natural language - and providing a minimum working example that reproduces it. -- All PRs must be written idiomatically: - - for Go: formatted according to - [gofmt](https://golang.org/cmd/gofmt/), and without any warnings - from [go lint](https://github.com/golang/lint) nor [go - vet](https://golang.org/cmd/vet/) - - for other languages, similar constraints apply. -- They should in general include tests, and those shall pass. - - If the PR is a bug fix, it has to include a new unit test that - fails before the patch is merged. - - If the PR is a new feature, it has to come with a suite of unit - tests, that tests the new functionality. - - In any case, all the PRs have to pass the personal evaluation of - at least one of the [maintainers](MAINTAINERS) of the project. - -### Getting started - -If you are a new contributor to the project, reading -[ARCHITECTURE.md](/ARCHITECTURE.md) is highly recommended, as it -contains all the details about the architecture of go-mysql-server and -its components. +go-mysql-server. We also welcome new issues with steps to reproduce a +problem. + +- PRs should include tests. +- If the PR is a bug fix, it should include a new unit test that fails + before the patch is merged. +- If the PR is a new feature, should have unit tests of the new + functionality. +- All contributions should include at least one end-to-end test in the + `enginetest` package. Typically this is just a new query with + expected results added to one of the large files of such queries in + the `queries` package. + +If you're confused, look at merged PRs for examples. diff --git a/MAINTAINERS b/MAINTAINERS index c185b3190a..e8875848ac 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1,2 +1,3 @@ Zach Musgrave (@zachmu) - +Daylon Wilkins (@hydrocharged) +Max Hoffman (@max-hoffman) \ No newline at end of file diff --git a/README.md b/README.md index 74e1e39e1e..fd197c5e40 100644 --- a/README.md +++ b/README.md @@ -1,46 +1,37 @@ # go-mysql-server -**go-mysql-server** is a SQL engine which parses standard SQL (based -on MySQL syntax) and executes queries on data sources of your -choice. A simple in-memory database and table implementation are -provided, and you can query any data source you want by implementing a -few interfaces. - -**go-mysql-server** also provides a server implementation compatible -with the MySQL wire protocol. That means it is compatible with MySQL -ODBC, JDBC, or the default MySQL client shell interface. +**go-mysql-server** is a data-source agnostic SQL engine and server +which runs queries on data sources you provide, using the MySQL +dialect and wire protocol. A simple in-memory database implementation +is included, and you can query any data source you want by +implementing your own backend. [Dolt](https://www.doltdb.com), a SQL database with Git-style -versioning, is the main database implementation of this package. -[Check out](https://docs.dolthub.com/introduction/what-is-dolt) that project for reference implementations. Or, hop into the Dolt discord [here](https://discord.com/invite/RFwfYpu) if you want to talk to the [core developers](https://www.dolthub.com/team) behind GMS. +versioning, is the main production database implementation of this +package. [Check +out](https://docs.dolthub.com/introduction/what-is-dolt) that project +for reference a implementation. Or, hop into the Dolt discord +[here](https://discord.com/invite/RFwfYpu) if you want to talk to the +[core developers](https://www.dolthub.com/team) behind +**go-mysql-server** and Dolt. + +## Compatibility + +With the exception of specific limitations (see below), +**go-mysql-server** is a drop-in replacement for MySQL. Any client +library, tool, query, SQL syntax, SQL function, etc. that works with +MySQL should also work with **go-mysql-server**. If you find a gap in +functionality, please file an issue. ## Scope of this project -These are the goals of **go-mysql-server**: - -- Be a generic extensible SQL engine that performs queries on your +- SQL server and engine to query your data sources. +- In-memory database implementation suitable for use in tests. +- Interfaces you can use to implement new backends to query your own data sources. -- Provide a simple database implementation suitable for use in tests. -- Define interfaces you can implement to query your own data sources. -- Provide a runnable server speaking the MySQL wire protocol, - connected to data sources of your choice. -- Optimize query plans. -- Allow implementors to add their own analysis steps and - optimizations. -- Support indexed lookups and joins on data tables that support them. -- Support external index driver implementations such as pilosa. -- With few caveats and using a full database implementation, be a +- With a few caveats and using a full database implementation, a drop-in MySQL database replacement. -Non-goals of **go-mysql-server**: - -- Be an application/server you can use directly. -- Provide any kind of backend implementation (other than the `memory` - one used for testing) such as json, csv, yaml. That's for clients to - implement and use. - -What's the use case of **go-mysql-server**? - **go-mysql-server** has two primary uses case: 1. Stand-in for MySQL in a golang test environment, using the built-in @@ -52,239 +43,20 @@ What's the use case of **go-mysql-server**? ## Installation -The import path for the package is `github.com/dolthub/go-mysql-server`. - -To install it, run: - -``` -go install github.com/dolthub/go-mysql-server@latest -``` - -Or to add it as a dependency to your project, in the directory with the go.mod file, run: +Add **go-mysql-server** as a dependency to your project. In the +directory with the `go.mod` file, run: ``` go get github.com/dolthub/go-mysql-server@latest ``` -## Go Documentation - -* [go-mysql-server godoc](https://godoc.org/github.com/dolthub/go-mysql-server) - -## SQL syntax - -The goal of **go-mysql-server** is to support 100% of the statements -that MySQL does. We are continuously adding more functionality to the -engine, but not everything is supported yet. To see what is currently -included check the [SUPPORTED](./SUPPORTED.md) file. - -## Third-party clients - -We support and actively test against certain third-party clients to -ensure compatibility between them and go-mysql-server. You can check -out the list of supported third party clients in the -[SUPPORTED_CLIENTS](./SUPPORTED_CLIENTS.md) file along with some -examples on how to connect to go-mysql-server using them. - -## Available functions - - -| Name | Description | -|:---------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------| -| `ABS(expr)` | Returns the absolute value of an expression.| -| `ACOS(expr)` | Returns the arccos of an expression.| -| `ARRAY_LENGTH(expr)` | If the json representation is an array, this function returns its size.| -| `ASCII(expr)` | Returns the numeric value of the leftmost character.| -| `ASIN(expr)` | Returns the arcsin of an expression.| -| `ATAN(expr)` | Returns the arctan of an expression.| -| `AVG(expr)` | Returns the average value of expr in all rows.| -| `BIN(expr)` | Returns the binary representation of a number.| -| `BIN_TO_UUID(...)` | Converts a binary UUID to a string UUID and returns the result.| -| `BIT_LENGTH(expr)` | Returns the data length of the argument in bits.| -| `CEIL(expr)` | Returns the smallest integer value that is greater than or equal to number.| -| `CEILING(expr)` | Returns the smallest integer value that is greater than or equal to number.| -| `CHAR_LENGTH(expr)` | Returns the length of the string in characters.| -| `CHARACTER_LENGTH(expr)` | Returns the length of the string in characters.| -| `COALESCE(...)` | Returns the first non-null value in a list.| -| `CONCAT(...)` | Concatenates any group of fields into a single string.| -| `CONCAT_WS(...)` | Concatenates any group of fields into a single string. The first argument is the separator for the rest of the arguments. The separator is added between the strings to be concatenated. The separator can be a string, as can the rest of the arguments. If the separator is NULL, the result is NULL.| -| `CONNECTION_ID()` | Returns the current connection ID.| -| `CONVERT_TZ(expr1, expr2, expr3)` | Converts a datetime value dt from the time zone given by from_tz to the time zone given by to_tz and returns the resulting value.| -| `COS(expr)` | Returns the cosine of an expression.| -| `COT(expr)` | Returns the arctangent of an expression.| -| `COUNT(expr)` | Returns a count of the number of non-NULL values of expr in the rows retrieved by a SELECT statement.| -| `CRC32(expr)` | Returns the cyclic redundancy check value of a given string as a 32-bit unsigned value.| -| `CURDATE()` | Returns the current date.| -| `CURRENT_DATE()` | Returns the current date.| -| `CURRENT_TIME()` | Returns the current time.| -| `CURRENT_TIMESTAMP(...)` | Returns the current date and time.| -| `CURRENT_USER()` | Returns the authenticated user name and host name.| -| `CURTIME()` | Returns the current time.| -| `DATABASE()` | Returns the default (current) database name.| -| `DATE(expr)` | Returns the date part of the given date.| -| `DATE_ADD(...)` | Adds the interval to the given date.| -| `DATE_FORMAT(expr1, expr2)` | Format date as specified.| -| `DATE_SUB(...)` | Subtracts the interval from the given date.| -| `DATETIME(...)` | Returns a DATETIME value for the expression given (e.g. the string '2020-01-02').| -| `DAY(expr)` | Returns the day of the month (0-31).| -| `DAYNAME(expr)` | Returns the name of the weekday.| -| `DAYOFMONTH(expr)` | Returns the day of the month (0-31).| -| `DAYOFWEEK(expr)` | Returns the day of the week of the given date.| -| `DAYOFYEAR(expr)` | Returns the day of the year of the given date.| -| `DEGREES(expr)` | Returns the number of degrees in the radian expression given.| -| `EXPLODE(expr)` | Generates a new row in the result set for each element in the expressions provided.| -| `FIRST(expr)` | Returns the first value in a sequence of elements of an aggregation.| -| `FIRST_VALUE(expr)` | Returns value of argument from first row of window frame.| -| `FLOOR(expr)` | Returns the largest integer value that is less than or equal to number.| -| `FORMAT(...)` | Returns a number formatted to specified number of decimal places.| -| `FOUND_ROWS()` | For a SELECT with a LIMIT clause, returns the number of rows that would be returned were there no LIMIT clause.| -| `FROM_BASE64(expr)` | Decodes the base64-encoded string str.| -| `FROM_UNIXTIME(expr)` | Formats Unix timestamp as a date.| -| `GET_LOCK(expr1, expr2)` | Gets a named lock.| -| `GREATEST(...)` | Returns the greatest numeric or string value.| -| `GROUP_CONCAT()` | Returns a string result with the concatenated non-NULL values from a group.| -| `HEX(expr)` | Returns the hexadecimal representation of the string or numeric value.| -| `HOUR(expr)` | Returns the hours of the given date.| -| `IF(expr1, expr2, expr3)` | If expr1 evaluates to true, retuns expr2. Otherwise returns expr3.| -| `IFNULL(expr1, expr2)` | If expr1 is not NULL, it returns expr1; otherwise it returns expr2.| -| `INET6_ATON(expr)` | Returns the numeric value of an IPv6 address.| -| `INET6_NTOA(expr)` | Returns the IPv6 address from a numeric value.| -| `INET_ATON(expr)` | Returns the numeric value of an IP address.| -| `INET_NTOA(expr)` | Returns the IP address from a numeric value.| -| `INSTR(expr1, expr2)` | Returns the 1-based index of the first occurence of str2 in str1, or 0 if it does not occur.| -| `IS_BINARY(expr)` | Returns whether a blob is a binary file or not.| -| `IS_FREE_LOCK(expr)` | Returns whether the named lock is free.| -| `IS_IPV4(expr)` | Returns whether argument is an IPv4 address.| -| `IS_IPV4_COMPAT(expr)` | Returns whether argument is an IPv4-compatible address.| -| `IS_IPV4_MAPPED(expr)` | Returns whether argument is an IPv4-mapped address.| -| `IS_IPV6(expr)` | Returns whether argument is an IPv6 address.| -| `IS_USED_LOCK(expr)` | Returns whether the named lock is in use; return connection identifier if true.| -| `IS_UUID(expr)` | Returns whether argument is a valid UUID.| -| `ISNULL(expr)` | Returns whether a expr is null or not.| -| `LAST(expr)` | Returns the last value in a sequence of elements of an aggregation.| -| `LAST_INSERT_ID()` | Returns value of the AUTOINCREMENT column for the last INSERT.| -| `LCASE(expr)` | Returns the string str with all characters in lower case.| -| `LEAST(...)` | Returns the smaller numeric or string value.| -| `LEFT(expr1, expr2)` | Returns the first N characters in the string given.| -| `LENGTH(expr)` | Returns the length of the string in bytes.| -| `LN(expr)` | Returns the natural logarithm of X.| -| `LOAD_FILE(expr)` | Returns a LoadFile object.| -| `LOG(...)` | If called with one parameter, this function returns the natural logarithm of X. If called with two parameters, this function returns the logarithm of X to the base B. If X is less than or equal to 0, or if B is less than or equal to 1, then NULL is returned.| -| `LOG10(expr)` | Returns the base-10 logarithm of X.| -| `LOG2(expr)` | Returns the base-2 logarithm of X.| -| `LOWER(expr)` | Returns the string str with all characters in lower case.| -| `LPAD(...)` | Returns the string str, left-padded with the string padstr to a length of len characters.| -| `LTRIM(expr)` | Returns the string str with leading space characters removed.| -| `MAX(expr)` | Returns the maximum value of expr in all rows.| -| `MD5(expr)` | Calculates MD5 checksum.| -| `MICROSECOND(expr)` | Returns the microseconds from argument.| -| `MID(...)` | Returns a substring from the provided string starting at pos with a length of len characters. If no len is provided, all characters from pos until the end will be taken.| -| `MIN(expr)` | Returns the minimum value of expr in all rows.| -| `MINUTE(expr)` | Returns the minutes of the given date.| -| `MONTH(expr)` | Returns the month of the given date.| -| `MONTHNAME(expr)` | Returns the name of the month.| -| `NOW(...)` | Returns the current timestamp.| -| `NULLIF(expr1, expr2)` | Returns NULL if expr1 = expr2 is true, otherwise returns expr1.| -| `PERCENT_RANK()` | Returns percentage rank value.| -| `POW(expr1, expr2)` | Returns the value of X raised to the power of Y.| -| `POWER(expr1, expr2)` | Returns the value of X raised to the power of Y.| -| `RADIANS(expr)` | Returns the radian value of the degrees argument given.| -| `RAND(...)` | Returns a random number in the range 0 <= x < 1. If an argument is given, it is used to seed the random number generator.| -| `REGEXP_LIKE(...)` | Returns whether string matches regular expression.| -| `REGEXP_REPLACE(...)` | Replaces substrings matching regular expression.| -| `RELEASE_ALL_LOCKS()` | Release all current named locks.| -| `RELEASE_LOCK(expr)` | Release the named lock.| -| `REPEAT(expr1, expr2)` | Returns a string consisting of the string str repeated count times.| -| `REPLACE(expr1, expr2, expr3)` | Returns the string str with all occurrences of the string from_str replaced by the string to_str.| -| `REVERSE(expr)` | Returns the string str with the order of the characters reversed.| -| `RIGHT(expr1, expr2)` | Returns the specified rightmost number of characters.| -| `ROUND(...)` | Rounds the number to decimals decimal places.| -| `ROW_COUNT()` | Returns the number of rows updated.| -| `ROW_NUMBER()` | Returns the number of rows updated.| -| `RPAD(...)` | Returns the string str, right-padded with the string padstr to a length of len characters.| -| `RTRIM(expr)` | Returns the string str with trailing space characters removed.| -| `SCHEMA()` | Returns the default (current) database name.| -| `SECOND(expr)` | Returns the seconds of the given date.| -| `SHA(expr)` | Calculates an SHA-1 160-bit checksum.| -| `SHA1(expr)` | Calculates an SHA-1 160-bit checksum.| -| `SHA2(expr1, expr2)` | Calculates an SHA-2 checksum.| -| `SIGN(expr)` | Returns the sign of the argument.| -| `SIN(expr)` | Returns the sine of the expression given.| -| `SLEEP(expr)` | Waits for the specified number of seconds (can be fractional).| -| `SOUNDEX(expr)` | Returns the soundex of a string.| -| `SPLIT(expr1, expr2)` | Returns the parts of the string str split by the separator sep as a JSON array of strings.| -| `SQRT(expr)` | Returns the square root of a nonnegative number X.| -| `STR_TO_DATE(...)` | Parses the date/datetime/timestamp expression according to the format specifier.| -| `STRCMP(expr1, expr2)` | Returns 0 if the strings are equal, -1 if the first expression is smaller, 1 otherwise. | -| `SUBSTR(...)` | Returns a substring from the provided string starting at pos with a length of len characters. If no len is provided, all characters from pos until the end will be taken.| -| `SUBSTRING(...)` | Returns a substring from the provided string starting at pos with a length of len characters. If no len is provided, all characters from pos until the end will be taken.| -| `SUBSTRING_INDEX(expr1, expr2, expr3)` | Returns a substring after count appearances of delim. If count is negative, counts from the right side of the string.| -| `SUM(expr)` | Returns the sum of expr in all rows.| -| `TAN(expr)` | Returns the tangent of the expression given.| -| `TIME_TO_SEC(expr)` | Returns the argument converted to seconds.| -| `TIMEDIFF(expr1, expr2)` | Returns expr1 − expr2 expressed as a time value. expr1 and expr2 are time or date-and-time expressions, but both must be of the same type.| -| `TIMESTAMP(...)` | Returns a timestamp value for the expression given (e.g. the string '2020-01-02').| -| `TO_BASE64(expr)` | Encodes the string str in base64 format.| -| `UCASE(expr)` | Converts string to uppercase.| -| `UNHEX(expr)` | Returns a string containing hex representation of a number.| -| `UNIX_TIMESTAMP(...)` | Returns the datetime argument to the number of seconds since the Unix epoch. With no argument, returns the number of seconds since the Unix epoch for the current time.| -| `UPPER(expr)` | Converts string to uppercase.| -| `USER()` | Returns the authenticated user name and host name.| -| `UTC_TIMESTAMP(...)` | Returns the current UTC timestamp.| -| `UUID()` | Returns a Universal Unique Identifier (UUID).| -| `UUID_TO_BIN(...)` | Converts string UUID to binary.| -| `VALUES(expr)` | Defines the values to be used during an INSERT.| -| `WEEK(...)` | Returns the week number.| -| `WEEKDAY(expr)` | Returns the weekday of the given date.| -| `WEEKOFYEAR(expr)` | Returns the calendar week of the date (1-53).| -| `YEAR(expr)` | Returns the year of the given date.| -| `YEARWEEK(...)` | Returns year and week for a date. The year in the result may be different from the year in the date argument for the first and the last week of the year.| - - -## Configuration - -The behaviour of certain parts of go-mysql-server can be configured -using either environment variables or session variables. - -Session variables are set using the following SQL queries: - -```sql -SET = -``` - - -| Name | Type | Description | -|:-----|:-----|:------------| -|`INMEMORY_JOINS`|environment|If set it will perform all joins in memory. Default is off.| -|`inmemory_joins`|session|If set it will perform all joins in memory. Default is off. This has precedence over `INMEMORY_JOINS`.| -|`MAX_MEMORY`|environment|The maximum number of memory, in megabytes, that can be consumed by go-mysql-server. Any in-memory caches or computations will no longer try to use memory when the limit is reached. Note that this may cause certain queries to fail if there is not enough memory available, such as queries using DISTINCT, ORDER BY or GROUP BY with groupings.| -|`DEBUG_ANALYZER`|environment|If set, the analyzer will print debug messages. Default is off.| - - -## Example +## Using the in-memory test server -`go-mysql-server` contains a SQL engine and server implementation. So, -if you want to start a server, first instantiate the engine and pass -your `sql.Database` implementation. - -It will be in charge of handling all the logic to retrieve the data -from your source. Here you can see an example using the in-memory -database implementation: +The in-memory test server can replace a real MySQL server in +tests. Start the server using the code in the [_example +directory](_example/main.go), also reproduced below. ```go -package main - -import ( - "fmt" - "time" - - "github.com/dolthub/go-mysql-server/sql/information_schema" - - sqle "github.com/dolthub/go-mysql-server" - "github.com/dolthub/go-mysql-server/memory" - "github.com/dolthub/go-mysql-server/server" - "github.com/dolthub/go-mysql-server/sql" -) - var ( dbName = "mydb" tableName = "mytable" @@ -295,20 +67,27 @@ var ( func main() { ctx := sql.NewEmptyContext() engine := sqle.NewDefault( - sql.NewDatabaseProvider( + memory.NewDBProvider( createTestDatabase(ctx), - information_schema.NewInformationSchemaDatabase(), )) + + if enableUsers { + if err := enableUserAccounts(ctx, engine); err != nil { + panic(err) + } + } config := server.Config{ Protocol: "tcp", Address: fmt.Sprintf("%s:%d", address, port), } - s, err := server.NewDefaultServer(config, engine) + + s, err := server.NewDefaultServer(config, engine) if err != nil { panic(err) } - if err = s.Start(); err != nil { + + if err = s.Start(); err != nil { panic(err) } } @@ -332,7 +111,13 @@ func createTestDatabase(ctx *sql.Context) *memory.Database { } ``` -Then, you can connect to the server with any MySQL client: +This example populates the database by creating `memory.Database` and +`memory.Table` objects via golang code, but you can also populate it +by issuing `CREATE DATABASE`, `CREATE TABLE`, etc. statements to the +server once it's running. + +Once the server is running, connect with any MySQL client, including +the golang MySQL connector and the `mysql` shell. ```bash > mysql --host=localhost --port=3306 --user=root mydb --execute="SELECT * FROM mytable;" @@ -346,256 +131,48 @@ Then, you can connect to the server with any MySQL client: +----------+-------------------+-------------------------------+----------------------------+ ``` -See the complete example [here](_example/main.go). - -### Queries examples +## Limitations of the in-memory database implementation -``` -SELECT count(name) FROM mytable -+---------------------+ -| COUNT(mytable.name) | -+---------------------+ -| 4 | -+---------------------+ -SELECT name,year(created_at) FROM mytable -+----------+--------------------------+ -| name | YEAR(mytable.created_at) | -+----------+--------------------------+ -| John Doe | 2018 | -| John Doe | 2018 | -| Jane Doe | 2018 | -| Evil Bob | 2018 | -+----------+--------------------------+ -SELECT email FROM mytable WHERE name = 'Evil Bob' -+-------------------+ -| email | -+-------------------+ -| evilbob@gmail.com | -+-------------------+ -``` +The in-memory database implementation included with this package is +intended for use in tests. It has specific limitations that we know +of: -## Custom data source implementation - -To create your own data source implementation you need to implement -the following interfaces: - -- `sql.Database` interface. This interface will provide tables from - your data source. You can also implement other interfaces on your - database to unlock additional functionality: - - `sql.TableCreator` to support creating new tables - - `sql.TableDropper` to support dropping tables - - `sql.TableRenamer` to support renaming tables - - `sql.ViewCreator` to support creating persisted views on your tables - - `sql.ViewDropper` to support dropping persisted views - -- `sql.Table` interface. This interface will provide rows of values - from your data source. You can also implement other interfaces on - your table to unlock additional functionality: - - `sql.InsertableTable` to allow your data source to be updated with - `INSERT` statements. - - `sql.UpdateableTable` to allow your data source to be updated with - `UPDATE` statements. - - `sql.DeletableTable` to allow your data source to be updated with - `DELETE` statements. - - `sql.ReplaceableTable` to allow your data source to be updated with - `REPLACE` statements. - - `sql.AlterableTable` to allow your data source to have its schema - modified by adding, dropping, and altering columns. - - `sql.IndexedTable` to declare your table's native indexes to speed - up query execution. - - `sql.IndexAlterableTable` to accept the creation of new native - indexes. - - `sql.ForeignKeyAlterableTable` to signal your support of foreign - key constraints in your table's schema and data. - - `sql.ProjectedTable` to return rows that only contain a subset of - the columns in the table. This can make query execution faster. - - `sql.FilteredTable` to filter the rows returned by your table to - those matching a given expression. This can make query execution - faster (if your table implementation can filter rows more - efficiently than checking an expression on every row in a table). - -You can see a really simple data source implementation in the `memory` -package. - -## Testing your data source implementation - -**go-mysql-server** provides a suite of engine tests that you can use -to validate that your implementation works as expected. See the -`enginetest` package for details and examples. - -## Indexes - -`go-mysql-server` exposes a series of interfaces to allow you to -implement your own indexes so you can speed up your queries. - -## Native indexes - -Tables can declare that they support native indexes, which means that -they support efficiently returning a subset of their rows that match -an expression. The `memory` package contains an example of this -behavior, but please note that it is only for example purposes and -doesn't actually make queries faster (although we could change this in -the future). - -Integrators should implement the `sql.IndexedTable` interface to -declare which indexes their tables support and provide a means of -returning a subset of the rows based on an `sql.IndexLookup` provided -by their `sql.Index` implementation. There are a variety of extensions -to `sql.Index` that can be implemented, each of which unlocks -additional capabilities: - -- `sql.Index`. Base-level interface, supporting equality lookups for - an index. -- `sql.AscendIndex`. Adds support for `>` and `>=` indexed lookups. -- `sql.DescendIndex`. Adds support for `<` and `<=` indexed lookups. -- `sql.NegateIndex`. Adds support for negating other index lookups. -- `sql.MergeableIndexLookup`. Adds support for merging two - `sql.IndexLookup`s together to create a new one, representing `AND` - and `OR` expressions on indexed columns. - -## Custom index driver implementation - -Index drivers provide different backends for storing and querying -indexes, without the need for a table to store and query its own -native indexes. To implement a custom index driver you need to -implement a few things: - -- `sql.IndexDriver` interface, which will be the driver itself. Not - that your driver must return an unique ID in the `ID` method. This - ID is unique for your driver and should not clash with any other - registered driver. It's the driver's responsibility to be fault - tolerant and be able to automatically detect and recover from - corruption in indexes. -- `sql.Index` interface, returned by your driver when an index is - loaded or created. -- `sql.IndexValueIter` interface, which will be returned by your - `sql.IndexLookup` and should return the values of the index. -- Don't forget to register the index driver in your `sql.Context` - using `context.RegisterIndexDriver(mydriver)` to be able to use it. - -To create indexes using your custom index driver you need to use -extension syntax `USING driverid` on the index creation statement. For -example: - -```sql -CREATE INDEX foo ON table USING driverid (col1, col2) -``` +- Not threadsafe. To avoid concurrency issues, limit DDL and DML + statements (`CREATE TABLE`, `INSERT`, etc.) to a single goroutine. +- No transaction support. Statements like `START TRANSACTION`, + `ROLLBACK`, and `COMMIT` are no-ops. +- Non-performant index implementation. Indexed lookups and joins + perform full table scans on the underlying tables. -go-mysql-server does not provide a production index driver -implementation. We previously provided a pilosa implementation, but -removed it due to the difficulty of supporting it on all platforms -(pilosa doesn't work on Windows). +## Custom backend implementations -You can see an example of a driver implementation in the memory -package. +You can create your own backend to query your own data sources by +implementing some interfaces. For detailed instructions, see the +[backend guide](./BACKEND.md). -### Metrics +## Technical documentation for contributors and backend developers -`go-mysql-server` utilizes `github.com/go-kit/kit/metrics` module to -expose metrics (counters, gauges, histograms) for certain packages (so -far for `engine`, `analyzer`, `regex`). If you already have -metrics server (prometheus, statsd/statsite, influxdb, etc.) and you -want to gather metrics also from `go-mysql-server` components, you -will need to initialize some global variables by particular -implementations to satisfy following interfaces: - -```go -// Counter describes a metric that accumulates values monotonically. -type Counter interface { - With(labelValues ...string) Counter - Add(delta float64) -} -// Gauge describes a metric that takes specific values over time. -type Gauge interface { - With(labelValues ...string) Gauge - Set(value float64) - Add(delta float64) -} -// Histogram describes a metric that takes repeated observations of the same -// kind of thing, and produces a statistical summary of those observations, -// typically expressed as quantiles or buckets. -type Histogram interface { - With(labelValues ...string) Histogram - Observe(value float64) -} -``` - -You can use one of `go-kit` implementations or try your own. For -instance, we want to expose metrics for _prometheus_ server. So, -before we start _mysql engine_, we have to set up the following -variables: - -```go -import( - "github.com/go-kit/kit/metrics/prometheus" - promopts "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/client_golang/prometheus/promhttp" -) -//.... -// engine metrics -sqle.QueryCounter = prometheus.NewCounterFrom(promopts.CounterOpts{ - Namespace: "go_mysql_server", - Subsystem: "engine", - Name: "query_counter", - }, []string{ - "query", - }) -sqle.QueryErrorCounter = prometheus.NewCounterFrom(promopts.CounterOpts{ - Namespace: "go_mysql_server", - Subsystem: "engine", - Name: "query_error_counter", -}, []string{ - "query", - "error", -}) -sqle.QueryHistogram = prometheus.NewHistogramFrom(promopts.HistogramOpts{ - Namespace: "go_mysql_server", - Subsystem: "engine", - Name: "query_histogram", -}, []string{ - "query", - "duration", -}) -// analyzer metrics -analyzer.ParallelQueryCounter = prometheus.NewCounterFrom(promopts.CounterOpts{ - Namespace: "go_mysql_server", - Subsystem: "analyzer", - Name: "parallel_query_counter", -}, []string{ - "parallelism", -}) -// regex metrics -regex.CompileHistogram = prometheus.NewHistogramFrom(promopts.HistogramOpts{ - Namespace: "go_mysql_server", - Subsystem: "regex", - Name: "compile_histogram", -}, []string{ - "regex", - "duration", -}) -regex.MatchHistogram = prometheus.NewHistogramFrom(promopts.HistogramOpts{ - Namespace: "go_mysql_server", - Subsystem: "regex", - Name: "match_histogram", -}, []string{ - "string", - "duration", -}) -``` - -One _important note_ - internally we set some _labels_ for metrics, -that's why have to pass those keys like "duration", "query", "driver", -... when we register metrics in `prometheus`. Other systems may have -different requirements. +- [Architecture](./ARCHITECTURE.md) is an overview of the various + packages of the project and how they fit together. +- [Contribution guide](./CONTRIBUTING.md) for new contributors, + including instructions for how to get your PR merged. ## Powered by go-mysql-server * [dolt](https://github.com/dolthub/dolt) * [gitbase](https://github.com/src-d/gitbase) (defunct) +Are you building a database backend using **go-mysql-server**? We +would like to hear from you and include you in this list. + ## Acknowledgements -**go-mysql-server** was originally developed by the {source-d} organzation, and this repository was originally forked from [src-d](https://github.com/src-d/go-mysql-server). We want to thank the entire {source-d} development team for their work on this project, especially Miguel Molina (@erizocosmico) and Juanjo Álvarez Martinez (@juanjux). +**go-mysql-server** was originally developed by the `{source-d}` +organzation, and this repository was originally forked from +[src-d](https://github.com/src-d/go-mysql-server). We want to thank +the entire `{source-d}` development team for their work on this +project, especially Miguel Molina (@erizocosmico) and Juanjo Álvarez +Martinez (@juanjux). ## License diff --git a/SUPPORTED.md b/SUPPORTED.md deleted file mode 100644 index 3b5b020345..0000000000 --- a/SUPPORTED.md +++ /dev/null @@ -1,201 +0,0 @@ -# Supported SQL Syntax - -**go-mysql-server**'s goal is to be a drop-in replacement for MySQL, -with every query and statement that works in MySQL behaving -identically in **go-mysql-server**. For most syntax and technical -questions, you should feel free to refer to the MySQL user -manual. Please file issues with any incompatibilities you discover. - -## Supported types - -- BOOLEAN -- INTEGER -- TINYINT -- SMALLINT -- MEDIUMINT -- INT -- BIGINT -- DECIMAL -- FLOAT -- DOUBLE -- BIT -- DATE -- TIME -- DATETIME -- TIMESTAMP -- YEAR -- CHAR -- VARCHAR -- BINARY -- VARBINARYX -- BLOB -- TINYTEXT -- TEXT -- MEDIUMTEXT -- LONGTEXT -- ENUM -- SET -- JSON - -## Data manipulation statements - -- DELETE -- INSERT -- REPLACE -- SELECT -- SUBQUERIES -- UPDATE - -## Data definition statements - -- ADD COLUMN -- ALTER COLUMN -- ALTER TABLE -- CHANGE COLUMN -- CREATE INDEX -- CREATE TABLE -- CREATE VIEW -- DESCRIBE TABLE -- DROP COLUMN -- DROP INDEX -- DROP TABLE -- DROP VIEW -- MODIFY COLUMN -- RENAME COLUMN -- SHOW CREATE TABLE -- SHOW CREATE VIEW -- SHOW DATABASES -- SHOW SCHEMAS -- SHOW TABLES - -## Transactional statements - -- BEGIN -- COMMIT -- LOCK TABLES -- START TRANSACTION -- UNLOCK TABLES - -## Session management statements - -- SET - -## Utility statements - -- EXPLAIN -- USE - -## Standard expressions - -- WHERE -- HAVING -- LIMIT -- OFFSET -- GROUP BY -- ORDER BY -- DISTINCT -- ALL -- AND -- NOT -- OR -- IF -- CASE / WHEN -- NULLIF -- COALESCE -- IFNULL -- LIKE -- IN / NOT IN -- IS NULL / IS NOT NULL -- INTERVAL -- Scalar subqueries -- Column ordinal references (standard MySQL extension) - -## Comparison expressions -- != -- == -- \> -- < -- \>= -- <= -- BETWEEN -- IN -- NOT IN -- REGEXP -- IS NOT NULL -- IS NULL - -## Aggregate functions - -- AVG -- COUNT and COUNT(DISTINCT) -- MAX -- MIN -- SUM (always returns DOUBLE) - -## Join expressions - -- CROSS JOIN -- INNER JOIN -- LEFT INNER JOIN -- RIGHT INNER JOIN -- NATURAL JOIN - -## Arithmetic expressions - -- \+ (including between dates and intervals) -- \- (including between dates and intervals) -- \* -- \/ -- << -- \>> -- & -- \| -- ^ -- div -- % - -## Subqueries - -Supported both as a table and as expressions but they can't access the -parent query scope. - -## Functions - -See README.md for the list of supported functions. - -# Notable limitations - -The engine is missing many features. The most important ones are noted -below. Our goal over time is 100% compatibility, which means adding -support for the items in this list. - -Some features are relatively easy to support, some are more -difficult. Please browse / file issues explaining your use case to -make your case for prioritizing missing features, or feel free to -discuss an implementation plan with us and submit a PR. - -## Missing features - -- Prepared statements / Execute -- Outer joins -- `AUTO INCREMENT` -- Transaction snapshotting / rollback -- Check constraint -- Window functions -- Common table expressions (CTEs) -- Stored procedures -- Events -- Cursors -- Triggers -- Users / privileges / `GRANT` / `REVOKE` (via SQL) -- `CREATE TABLE AS` -- `DO` -- `HANDLER` -- `IMPORT TABLE` -- `LOAD DATA` / `LOAD XML` -- `SELECT FOR UPDATE` -- `TABLE` (alternate select syntax) -- `TRUNCATE` -- Alter index -- Alter view -- Create function diff --git a/_example/main.go b/_example/main.go index cfbb1747a6..844c68515b 100644 --- a/_example/main.go +++ b/_example/main.go @@ -51,9 +51,10 @@ var ( func main() { ctx := sql.NewEmptyContext() engine := sqle.NewDefault( - sql.NewDatabaseProvider( + memory.NewDBProvider( createTestDatabase(ctx), )) + // This variable may be found in the "users_example.go" file. Please refer to that file for a walkthrough on how to // set up the "mysql" database to allow user creation and user checking when establishing connections. This is set // to false for this example, but feel free to play around with it and see how it works.