NEAR Lake NFT indexer experiencing slow data retrieval

NFT
1

Hey everyone, I’m having some trouble with the NEAR Lake NFT indexer for mainnet block info. It’s super slow getting data. Sometimes the streamer_messages_queue.get() method takes forever (like hundreds of seconds) or just gets stuck.

I’m wondering if I messed up the config or if there are limits causing this slowdown. Here’s a snippet of what I’m doing:

async def fetch_block_data(config):
    data_stream, message_queue = initialize_streamer(config)
    while True:
        begin = time.time()
        print(f'Listening started at: {begin}')
        block_message = await message_queue.get()
        finish = time.time()
        print(f'Message retrieval took: {finish - begin} seconds')
        print(f'Block {block_message.block.header.height} - Shard count: {len(block_message.shards)}')
        await process_block_message(block_message)

# Output example:
# Listening started at: 1660711985
# Message retrieval took: 282 seconds
# Block 71516637 - Shard count: 4

The message_queue.get() part is where it’s getting stuck. Any ideas what could be causing this? Thanks!

2

hey there ClimbingMountain! :wave:

wow, that’s quite the wait time you’re experiencing! :open_mouth: have you tried checking your network connection? sometimes that can cause unexpected slowdowns.

i’m curious, what kind of hardware are you running this on? could it be a resource issue?

also, how many NFTs are you trying to index? if it’s a massive amount, that could definitely impact performance.

have you reached out to the NEAR devs directly? they might have some insider tips for optimizing the indexer.

let us know if you find a solution - i’m sure others would benefit from your experience too!

3

yo ClimbingMountain, that wait time is brutal! :grimacing: have u tried running it on a different machine or network? Sometimes my indexer acts up when i’m on crappy wifi. Also, double-check ur config - i once had a similar issue cuz i messed up some settings. good luck fixing it dude!

4

I’ve encountered similar issues with the NEAR Lake NFT indexer. In my experience, the problem often stems from network latency or insufficient system resources. Have you monitored your CPU and memory usage during indexing? It might be worth scaling up your hardware if you’re hitting resource limits.

Another possibility is that you’re encountering rate limiting from the NEAR RPC nodes. You could try implementing exponential backoff and retry logic to handle temporary network issues or API limits.

Lastly, ensure you’re using the latest version of the indexer library. There have been performance improvements in recent releases that might address your slowdown.

If the issue persists, I’d recommend profiling your code to identify any bottlenecks in your processing logic. Let us know if you make any progress resolving this.