Skip to content

Commit

Permalink
chore: remove orphaned pages (#2065)
Browse files Browse the repository at this point in the history
* remove orphan pages

* resolve broken links

* resolve near-lake.md broken links

* resolve indexer-for-explorer urls

* reintroduced maintenance-windows, resolved missing links

* fix

* Update docs/1.concepts/3.advanced/indexers.md

---------

Co-authored-by: Guille <[email protected]>
  • Loading branch information
flmel and gagdiez authored Jun 19, 2024
1 parent bf47b0b commit 9e140a3
Show file tree
Hide file tree
Showing 27 changed files with 18 additions and 797 deletions.
2 changes: 1 addition & 1 deletion blog/2024-06-05.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Scrolling down will reveal the option `Windows Subsystem for Linux`. Make sure i

#### Start WSL and Install Ubuntu

Now we will spend some time in either `PowerShell` or [Windows Terminal](https://aka.ms/terminal), which is a modern terminal application that supports various command-line tools and shells.
Now we will spend some time in either `PowerShell` or [Windows Terminal](https://apps.microsoft.com/detail/9n0dx20hk701), which is a modern terminal application that supports various command-line tools and shells.

```
wsl --install --d Ubuntu
Expand Down
7 changes: 3 additions & 4 deletions docs/1.concepts/3.advanced/indexers.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,16 +77,15 @@ To query data across many blocks requires the aggregation of results from multip

:::

Because indexers listen to the *stream of data* from the blockchain and the data can be immediately filtered and processed according to defined requirements, they can be used to simplify the "wide" queries execution. For example, a stream of data can be written to a permanent database for later data analysis using a convenient query language like SQL. That is what [Indexer for Explorer](/tools/indexer-for-explorer) is doing.
Because indexers listen to the *stream of data* from the blockchain and the data can be immediately filtered and processed according to defined requirements, they can be used to simplify the "wide" queries execution. For example, a stream of data can be written to a permanent database for later data analysis using a convenient query language like SQL.

Another example that highlights the need for a "wide query" is when you use a seed phrase to recover one or more accounts. Since a seed phrase essentially represents a signing key pair, the recovery is for all accounts that share the associated public key. Therefore, when a seed phrase is used to recover an account via [NEAR Wallet](https://wallet.near.org), the query requires that all accounts with a matching public key are found and recovered. [NEAR Indexer for Explorer](/tools/indexer-for-explorer) is storing this data in a permanent database and this allows [NEAR Wallet](https://wallet.near.org) to perform such "wide queries". This is impossible to achieve using JSON-RPC only.
Another example that highlights the need for a "wide query" is when you use a seed phrase to recover one or more accounts. Since a seed phrase essentially represents a signing key pair, the recovery is for all accounts that share the associated public key. Therefore, when a seed phrase is used to recover an account via [NEAR Wallet](https://wallet.near.org), the query requires that all accounts with a matching public key are found and recovered. Utilizing [Near Lake Framework](https://github.com/near/near-lake-framework-rs) can be used to store this data in a permanent database and this allows [NEAR Wallet](https://wallet.near.org) to perform such "wide queries". This is impossible to achieve using JSON-RPC only.

## Summary

We hope this article gives you an understanding of the Indexer concept. Also, we hope now you can easily decide whether you need an indexer for your application.

## What's next?

We encourage you to learn more about the [Lake Indexer project](/tools/realtime#near-lake-indexer). Please, proceed to [Tutorials](/build/data-infrastructure/lake-framework/near-lake-state-changes-indexer) section to learn how to build an indexer on practice.
We encourage you to learn more about the [Lake Indexer project](/build/data-infrastructure/lake-framework/near-lake). Please, proceed to [Tutorials](/build/data-infrastructure/lake-framework/near-lake-state-changes-indexer) section to learn how to build an indexer on practice.

Alternatively, there are a few other third-party indexers that are tightly integrated with the NEAR ecosystem. You can review all of your data sourcing options (including The Graph, Pagoda, Pipespeak, and SubQuery) under [data tools](/concepts/data-flow/data-storage#data-tools).
6 changes: 0 additions & 6 deletions docs/1.concepts/3.advanced/near-indexer-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,3 @@ These days, we use NEAR Indexer Framework to implement [NEAR Lake Indexer](https
See the [example](https://github.com/nearprotocol/nearcore/tree/master/tools/indexer/example) for further technical details.

- [`near-examples/indexer-tx-watcher-example`](https://github.com/near-examples/indexer-tx-watcher-example) NEAR Indexer example that watches for transaction for specified accounts/contracts

:::info NEAR Indexer Framework usage

The most famous project build on top of NEAR Indexer Framework is [NEAR Indexer for Explorer](/tools/indexer-for-explorer)

:::
2 changes: 1 addition & 1 deletion docs/1.concepts/3.advanced/near-lake-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ https://github.com/near/near-lake-framework/

## Description

NEAR Lake Framework is an ecosystem of library companions to [NEAR Lake](/tools/realtime#near-lake-indexer). They allow you to build your own indexer that subscribes to the stream of blocks from the [NEAR Lake](/tools/realtime#near-lake-indexer) data source and create your own logic to process the NEAR Protocol data in the programming languages of your choice (at the moment, there are implementations in [Python](http://pypi.org/project/near-lake-framework), [JavaScript](https://www.npmjs.com/package/near-lake-framework), and [Rust](https://crates.io/crates/near-lake-framework)).
NEAR Lake Framework is an ecosystem of library companions to [NEAR Lake](/build/data-infrastructure/lake-framework/near-lake). They allow you to build your own indexer that subscribes to the stream of blocks from the [NEAR Lake](/build/data-infrastructure/lake-framework/near-lake) data source and create your own logic to process the NEAR Protocol data in the programming languages of your choice (at the moment, there are implementations in [Python](http://pypi.org/project/near-lake-framework), [JavaScript](https://www.npmjs.com/package/near-lake-framework), and [Rust](https://crates.io/crates/near-lake-framework)).

:::tip NEAR Lake Framework announcement

Expand Down
6 changes: 0 additions & 6 deletions docs/1.concepts/3.advanced/virtual_machine.md

This file was deleted.

37 changes: 0 additions & 37 deletions docs/1.concepts/abstraction/mpc.md

This file was deleted.

55 changes: 0 additions & 55 deletions docs/1.concepts/basics/overview.md

This file was deleted.

20 changes: 0 additions & 20 deletions docs/1.concepts/basics/technical_stuff.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/1.concepts/data-flow/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ As an alternative, you can try one of the many available indexing services on th


* [NEAR Indexer Framework](/concepts/advanced/near-indexer-framework): a micro-framework providing you with a "live" stream of blocks. Useful to handle on-chain real-time "events".
* [NEAR Indexer for Explorer](/tools/indexer-for-explorer): leverages the indexer micro-framework to watch and store all of the blockchain's events/data into a transactional PostgreSQL database. You can clone the [GitHub repository](https://github.com/near/near-indexer-for-explorer) and customize your own indexer solution.
* [NEAR Indexer for Explorer](https://github.com/near/near-indexer-for-explorer): leverages the indexer micro-framework to watch and store all of the blockchain's events/data into a transactional PostgreSQL database. You can clone the [GitHub repository](https://github.com/near/near-indexer-for-explorer) and customize your own indexer solution.
* [NEAR Lake Framework](/concepts/advanced/near-lake-framework): a companion library to NEAR Lake. It allows you to build your own indexer that watches a stream of blocks from a NEAR Lake data source and allows you to create your own logic to process that data. Keep in mind this is probably the one you want to use for future projects, instead of the Indexer Framework. Read [why it is better](/concepts/advanced/near-indexer-framework#why-is-it-better-than-near-indexer-framework).
* [NEAR Lake Indexer](/concepts/advanced/near-lake-framework): leverages the indexer micro-framework to watch and store all of the blockchain's events/data as JSON files on a user-specified AWS S3 or S3-compatible storage.
* [Near Query API](https://near.org/dataplatform.near/widget/QueryApi.App): Query API allows you to seamlessly create, manage, and discover indexers on NEAR. Developers can deploy their indexers to aggregate historical data in a matter of minutes, while accessing the data via custom GraphQL queries.
Expand Down
2 changes: 1 addition & 1 deletion docs/1.concepts/web3/near.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ Natively, NEAR smart contracts store data as key-value pairs. This is quite limi
- While very useful, vectors, maps and sets won’t match the flexibility and power of classical relational databases. Even implementations of simple filtering and searching might be quite complex and require a lot of gas to execute, especially if multiple entities with relations between them are involved.
- They are limited to a single contract. If data from multiple contracts is required, aggregation should be performed using cross-contract calls or on a client side, which is quite expensive in terms of gas and time.

To support more complex data retrieval scenarios, smart contract data should be put in a more appropriate store, like a relational database. [Indexers](../../4.tools/indexer4explorer.md) are used to achieve this. In a nutshell, indexer is just a special kind of blockchain node that processes incoming transactions and puts relevant data into a database. Collected data can be exposed to a client using a simple API server (e.g. REST or GraphQL).
To support more complex data retrieval scenarios, smart contract data should be put in a more appropriate store, like a relational database. Indexers are used to achieve this. In a nutshell, indexer is just a special kind of blockchain node that processes incoming transactions and puts relevant data into a database. Collected data can be exposed to a client using a simple API server (e.g. REST or GraphQL).


![image](/docs/assets/web3/web3-15.png)
Expand Down
2 changes: 1 addition & 1 deletion docs/1.concepts/web3/nfts.md
Original file line number Diff line number Diff line change
Expand Up @@ -569,7 +569,7 @@ As we already determined, an indexing service is needed in order to support mark
</div>


While any technology of choice can be used to implement Database and API, an indexer itself is usually implemented using Rust, since a [framework is available](https://github.com/near/nearcore/tree/master/chain/indexer) for this language. Guide how to implement your own indexer can be found [here](../../4.tools/indexer4explorer.md).
While any technology of choice can be used to implement Database and API, an indexer itself is can be implemented using both [Rust](https://github.com/near/near-lake-framework-rs) and [JavaScript](https://github.com/near/near-lake-framework-js). See [this example](https://github.com/near-examples/near-lake-nft-indexer) written in Rust.

Usually, an indexer works by extracting data from [Events](https://nomicon.io/Standards/EventsFormat), which are basically just structured log messages written during contract execution.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ That's it. Now we can compile the code and run it

:::danger Credentials

To be able to access the data from [NEAR Lake](/tools/realtime#near-lake-indexer) you need to provide credentials. Please, see the [Credentials](../running-near-lake/credentials.md) article
To be able to access the data from [NEAR Lake](/build/data-infrastructure/lake-framework/near-lake) you need to provide credentials. Please, see the [Credentials](../running-near-lake/credentials.md) article

:::

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ You can stop the indexer by pressing CTRL+C

:::danger Credentials

To be able to access the data from [NEAR Lake](/tools/realtime#near-lake-indexer) you need to provide credentials. Please, see the [Credentials](../../lake-framework/running-near-lake/credentials.md) article
To be able to access the data from [NEAR Lake](/build/data-infrastructure/lake-framework/near-lake) you need to provide credentials. Please, see the [Credentials](../../lake-framework/running-near-lake/credentials.md) article

:::

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ sidebar_label: Lake Overview
title: NEAR Lake Indexer
---

NEAR Lake is an indexer built on top of [NEAR Indexer Framework](/concepts/advanced/near-indexer-framework) to watch the network and store all the [events](../../../4.tools/events.md) as JSON files on AWS S3.
NEAR Lake is an indexer built on top of [NEAR Indexer Framework](/concepts/advanced/near-indexer-framework) to watch the network and store all the event logs such as [FT Events](https://nomicon.io/Standards/Tokens/FungibleToken/Event) and [NFT Events](https://nomicon.io/Standards/Tokens/NonFungibleToken/Event) as JSON files on AWS S3.

:::info GitHub repo

Expand Down
Loading

0 comments on commit 9e140a3

Please sign in to comment.