Awesome
NEAR Lake Framework JS
Available in programming languages: Rust | Javascript
NEAR Lake Framework is a small library companion to NEAR Lake. It allows you to build your own indexer that subscribes to the stream of blocks from the NEAR Lake data source and create your own logic to process the NEAR Protocol data.
Official NEAR Lake Framework launch announcement has been published on the NEAR Gov Forum
Example
import { startStream, types } from '@near-lake/framework';
const lakeConfig: types.LakeConfig = {
s3BucketName: 'near-lake-data-mainnet',
s3RegionName: 'eu-central-1',
startBlockHeight: 66264389,
};
async function handleBlock(
block: types.Block,
): Promise<void> {
// custom logic for handling the block
let events = block.eventsByAccountId("x.paras.near")
console.log(events)
}
(async () => {
await startStream(lakeConfig, handleStreamerMessage);
})();
Packages
NEAR Lake Framework JS originally existed as a single library: near-lake-framework. This package is still avaiable on NPM and can be used as is. But to make package consumption easier, the library has since been split in to sub-packages:
- @near-lake/framework - core library for streaming blocks from NEAR Lake
- @near-lake/primitives - companion library to help interaction with blockchain data
In most cases you will only need @near-lake/framework
. For more information on both, see their respective READMEs.
Tutorial
Please, read the tutorial JavaScript NEAR Lake indexer basic tutorial
How to use
Custom S3 storage
In case you want to run your own near-lake instance and store data in some S3 compatible storage (Minio or Localstack as example)
You can owerride default S3 API endpoint by using s3_endpoint
option
- run minio
$ mkdir -p /data/near-lake-custom && minio server /data
- add
s3_endpoint
parameter to LakeConfig instance
const localEndpoint: types.EndpointConfig = {
protocol: 'http',
hostname: '0.0.0.0',
port: 9000,
path: '/',
};
const lakeConfig: types.LakeConfig = {
s3Endpoint: localEndpoint,
s3BucketName: "near-lake-custom",
s3RegionName: "eu-central-1",
s3ForcePathStyle: true,
startBlockHeight: 0,
};
AWS S3 Credentials
In order to be able to get objects from the AWS S3 bucket you need to provide the AWS credentials.
AWS default profile configuration with aws configure looks similar to the following:
~/.aws/credentials
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS docs: Configuration and credential file settings
Dependencies
Install near-lake-framework
$ npm i near-lake-framework
After that import in your code:
import { startStream, types } from 'near-lake-framework';
Configuration
Everything should be configured before the start of your indexer application via LakeConfig
struct.
Available parameters:
s3Endpoint: string
- provide the AWS S3 custom API ednpoints3BucketName: string
- provide the AWS S3 bucket name (near-lake-testnet
,near-lake-mainnet
or yours if you run your own NEAR Lake)s3RegionName: string
- provide the region for AWS S3 bucketstartBlockHeight: number
- block height to start the stream fromblocksPreloadPoolSize: number
- provide the number of blocks to preload (default: 200)
Cost estimates
TL;DR approximately $18.15 per month (for AWS S3 access, paid directly to AWS) for the reading of fresh blocks
Explanation:
Assuming NEAR Protocol produces accurately 1 block per second (which is really not, the average block production time is 1.3s). A full day consists of 86400 seconds, that's the max number of blocks that can be produced.
According the Amazon S3 prices list
requests are charged for $0.005 per 1000 requests and get
is charged for $0.0004 per 1000 requests.
Calculations (assuming we are following the tip of the network all the time):
86400 blocks per day * 5 requests for each block / 1000 requests * $0.0004 per 1k requests = $0.173 * 30 days = $5.19
Note: 5 requests for each block means we have 4 shards (1 file for common block data and 4 separate files for each shard)
And a number of list
requests we need to perform for 30 days:
86400 blocks per day / 1000 requests * $0.005 per 1k list requests = $0.432 * 30 days = $12.96
$5.19 + $12.96 = $18.15
The price depends on the number of shards
Future plans
The main NEAR Lake Framework library we develop is a Rust-lang version. The JS version is following the main one, so there is might be some delays in delivering fixes and features
We use Milestones with clearly defined acceptance criteria:
Running examples
Inside apps/
we have created a created a bunch of example implementations for using the indexer.
To test out any of the indexers, from the root, run:
$ npm run demo --name=<example_package_name>