Chia Seeder (Chia-Network#8991)

* initial hack * crawler * add pytz * Checkpoint. * Catch some bugs. * Localhost dig working. * Checkpoint: return only high quality nodes. * Statistics. * Try improving finding reliable nodes. * Bug. * Move db to memory. * Timestamp in the last 5 days. * Increase crawl parameters, 180+ connections per sec. * Bug. * Optimize for DNS traffic. * Prepare for hosting. * Minimum height. * Typo. * Try catch everything. * dnslib. * Add db, format code. * nits. * No connections for the dns server. * Rename src -> chia * Fix some issues with v1.1 * Crawler task pool. * Optimize closing connections. * Split crawler and dns server. * Install instructions. * Catch startup bug. * Try a big timeout for lock aquire. * lint. * Lint. * Initial commit extended stats. * Simplify code. * Config. * Correct stats. * Be more restrictive in crawling. * Attempt to fix stats bug. * Add other peers port to config. * Update README for the config. * Simplify crawl task. * Fix bug on restarts. * Prevent log spamming. * More spam prevention. * Fix bug. * Ipv6 (Chia-Network#1) * Enable ipv6. * Fix bug. * Use numeric codes for QTYPE. * ANY working. * More spam prevention. * Try to improve IPv6 selection. * Log IPv6 available. * Try to crawl more aggresive for v6. * rename dns.py to crawler_dns.py so it doesn't conflict with imported package names * Remove pytz package off dependencies * Tidy-up ws_connection.py * Fix spelling * Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler * More detailed info in the README wrt Chia Seeder * Nit * More memetic naming of Chia Seeder * Nit * Add entry points * Add entry in packages * Patch some methods on the upstream server * Update peer record fields * Standard library imports first * Crawler API check * Reconcile crawl store * Account for crawler_db_path in config * Await crawl store load DB and load reliable peers * Updates to crawler * Rename to dns_server * Crawler-specific overrides for the chia server * Edit comment * Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit * Nit * Update service groups * Expand name maps, mostly * Fix the init config * Remove unused import * total_records unused at this stage * Remove ios_reliable in peer_reliability table * Remove row[20] entry * Split overly long line * Fix * Type hint for ns_records * Reconcile mismatch btw type int and uint64 * Type annotations in crawler * Check whether crawl store is set * Remove upnp_list * Lint * Chia Seeder CLI * Lint * Two white spaces * 3rd party package import * Cleaner way to handle overrides for ChiaServer method * Address linter warnings * Rename * Nits * Fix * Change port # * Most chia_seeder commands up and running * Rename * Progress of sorts * Fix * Improve legibility * Fix naming * Fix setup.py * Lint * None -> '' * Remove whitespace * Rename * Log ipv6 better. (Chia-Network#9227) * Log ipv6 better. * Lint. * - * Undo GUI changes * Another attempt * GUI changes Co-authored-by: Yostra <[email protected]> Co-authored-by: Florin Chirica <[email protected]> Co-authored-by: Chris Marslender <[email protected]>
pfinazzo · Nov 28, 2021 · 025c45f · 025c45f
1 parent a80b7df
commit 025c45f
Show file tree

Hide file tree

Showing 20 changed files with 1,717 additions and 9 deletions.
diff --git a/README.md b/README.md
@@ -11,6 +11,7 @@
 Chia is a modern cryptocurrency built from scratch, designed to be efficient, decentralized, and secure. Here are some of the features and benefits:
 * [Proof of space and time](https://docs.google.com/document/d/1tmRIb7lgi4QfKkNaxuKOBHRmwbVlGL4f7EsBDr_5xZE/edit) based consensus which allows anyone to farm with commodity hardware
 * Very easy to use full node and farmer GUI and cli (thousands of nodes active on mainnet)
+* [Chia seeder](https://github.com/Chia-Network/chia-blockchain/wiki/Chia-Seeder-User-Guide), which maintains a list of reliable nodes within the Chia network via a built-in DNS server.
 * Simplified UTXO based transaction model, with small on-chain state
 * Lisp-style Turing-complete functional [programming language](https://chialisp.com/) for money related use cases
 * BLS keys and aggregate signatures (only one signature per block)
@@ -33,6 +34,7 @@ TCP port 8444 access to your peer.
 These methods tend to be router make/model specific.
 
 Most users should only install harvesters, farmers, plotter, full nodes, and wallets.
+Setting up a seeder is best left to more advanced users.
 Building Timelords and VDFs is for sophisticated users, in most environments.
 Chia Network and additional volunteers are running sufficient Timelords
 for consensus.

diff --git a/chia/cmds/seeder.py b/chia/cmds/seeder.py
@@ -0,0 +1,204 @@
+import os
+from pathlib import Path
+from typing import Dict
+
+import click
+
+import chia.cmds.configure as chia_configure
+from chia import __version__
+from chia.cmds.chia import monkey_patch_click
+from chia.cmds.init_funcs import init
+from chia.seeder.util.config import patch_default_seeder_config
+from chia.seeder.util.service_groups import all_groups, services_for_groups
+from chia.seeder.util.service import launch_service, kill_service
+from chia.util.config import load_config, save_config
+from chia.util.default_root import DEFAULT_ROOT_PATH
+
+CONTEXT_SETTINGS = dict(help_option_names=["-h", "--help"])
+
+
+@click.group(
+    help=f"\n  Manage the Chia Seeder ({__version__})\n",
+    epilog="Try 'chia seeder start crawler' or 'chia seeder start server'",
+    context_settings=CONTEXT_SETTINGS,
+)
+@click.option("--root-path", default=DEFAULT_ROOT_PATH, help="Config file root", type=click.Path(), show_default=True)
+@click.pass_context
+def cli(
+    ctx: click.Context,
+    root_path: str,
+) -> None:
+    from pathlib import Path
+
+    ctx.ensure_object(dict)
+    ctx.obj["root_path"] = Path(root_path)
+
+
+@cli.command("version", short_help="Show the Chia Seeder version")
+def version_cmd() -> None:
+    print(__version__)
+
+
+@click.command("init", short_help="Create or migrate the configuration")
+@click.pass_context
+def init_cmd(ctx: click.Context, **kwargs):
+    print("Calling Chia Seeder Init...")
+    init(None, ctx.obj["root_path"], True)
+    if os.environ.get("CHIA_ROOT", None) is not None:
+        print(f"warning, your CHIA_ROOT is set to {os.environ['CHIA_ROOT']}.")
+    root_path = ctx.obj["root_path"]
+    print(f"Chia directory {root_path}")
+    if root_path.is_dir() and not Path(root_path / "config" / "config.yaml").exists():
+        # This is reached if CHIA_ROOT is set, but there is no config
+        # This really shouldn't happen, but if we dont have the base chia config, we can't continue
+        print("Config does not exist. Can't continue!")
+        return -1
+    patch_default_seeder_config(root_path)
+    return 0
+
+
+@click.command("start", short_help="Start service groups")
+@click.argument("group", type=click.Choice(all_groups()), nargs=-1, required=True)
+@click.pass_context
+def start_cmd(ctx: click.Context, group: str) -> None:
+    services = services_for_groups(group)
+
+    for service in services:
+        print(f"Starting {service}")
+        launch_service(ctx.obj["root_path"], service)
+
+
+@click.command("stop", short_help="Stop service groups")
+@click.argument("group", type=click.Choice(all_groups()), nargs=-1, required=True)
+@click.pass_context
+def stop_cmd(ctx: click.Context, group: str) -> None:
+    services = services_for_groups(group)
+
+    for service in services:
+        print(f"Stopping {service}")
+        kill_service(ctx.obj["root_path"], service)
+
+
+def configure(
+    root_path: Path,
+    testnet: str,
+    crawler_db_path: str,
+    minimum_version_count: int,
+    domain_name: str,
+    nameserver: str,
+):
+    # Run the parent config, in case anything there (testnet) needs to be run, THEN load the config for local changes
+    chia_configure.configure(root_path, "", "", "", "", "", "", "", "", testnet, "")
+
+    config: Dict = load_config(DEFAULT_ROOT_PATH, "config.yaml")
+    change_made = False
+    if testnet is not None:
+        if testnet == "true" or testnet == "t":
+            print("Updating Chia Seeder to testnet settings")
+            port = 58444
+            network = "testnet7"
+            bootstrap = ["testnet-node.chia.net"]
+
+            config["seeder"]["port"] = port
+            config["seeder"]["other_peers_port"] = port
+            config["seeder"]["selected_network"] = network
+            config["seeder"]["bootstrap_peers"] = bootstrap
+
+            change_made = True
+
+        elif testnet == "false" or testnet == "f":
+            print("Updating Chia Seeder to mainnet settings")
+            port = 8444
+            network = "mainnet"
+            bootstrap = ["node.chia.net"]
+
+            config["seeder"]["port"] = port
+            config["seeder"]["other_peers_port"] = port
+            config["seeder"]["selected_network"] = network
+            config["seeder"]["bootstrap_peers"] = bootstrap
+
+            change_made = True
+        else:
+            print("Please choose True or False")
+
+    if crawler_db_path is not None:
+        config["seeder"]["crawler_db_path"] = crawler_db_path
+        change_made = True
+
+    if minimum_version_count is not None:
+        config["seeder"]["minimum_version_count"] = minimum_version_count
+        change_made = True
+
+    if domain_name is not None:
+        config["seeder"]["domain_name"] = domain_name
+        change_made = True
+
+    if nameserver is not None:
+        config["seeder"]["nameserver"] = nameserver
+        change_made = True
+
+    if change_made:
+        print("Restart any running Chia Seeder services for changes to take effect")
+        save_config(root_path, "config.yaml", config)
+    return 0
+
+
+@click.command("configure", short_help="Modify configuration")
+@click.option(
+    "--testnet",
+    "-t",
+    help="configures for connection to testnet",
+    type=click.Choice(["true", "t", "false", "f"]),
+)
+@click.option(
+    "--crawler-db-path",
+    help="configures for path to the crawler database",
+    type=str,
+)
+@click.option(
+    "--minimum-version-count",
+    help="configures how many of a particular version must be seen to be reported in logs",
+    type=int,
+)
+@click.option(
+    "--domain-name",
+    help="configures the domain_name setting. Ex: `seeder.example.com.`",
+    type=str,
+)
+@click.option(
+    "--nameserver",
+    help="configures the nameserver setting. Ex: `example.com.`",
+    type=str,
+)
+@click.pass_context
+def configure_cmd(
+    ctx,
+    testnet,
+    crawler_db_path,
+    minimum_version_count,
+    domain_name,
+    nameserver,
+):
+    configure(
+        ctx.obj["root_path"],
+        testnet,
+        crawler_db_path,
+        minimum_version_count,
+        domain_name,
+        nameserver,
+    )
+
+
+cli.add_command(init_cmd)
+cli.add_command(start_cmd)
+cli.add_command(stop_cmd)
+cli.add_command(configure_cmd)
+
+
+def main() -> None:
+    monkey_patch_click()
+    cli()  # pylint: disable=no-value-for-parameter
+
+
+if __name__ == "__main__":
+    main()
diff --git a/chia/daemon/server.py b/chia/daemon/server.py
@@ -100,6 +100,9 @@ class PlotEvent(str, Enum):
         "chia_timelord": "start_timelord",
         "chia_timelord_launcher": "timelord_launcher",
         "chia_full_node_simulator": "start_simulator",
+        "chia_seeder": "chia_seeder",
+        "chia_seeder_crawler": "chia_seeder_crawler",
+        "chia_seeder_dns": "chia_seeder_dns",
     }
 
     def executable_for_service(service_name: str) -> str:

diff --git a/chia/seeder/__init__.py b/chia/seeder/__init__.py