Why is the Near Lake NFT indexer so slow to fetch data?

NFT
1

Hey everyone, I’m having trouble with the Near Lake NFT indexer for mainnet block info. It’s super sluggish! Sometimes it takes forever (like hundreds of seconds) to get data in the streamer_messages_queue.get() method. Other times, it just gets stuck there.

I’m wondering if I messed up the config or if there are limits causing this snail-like speed. Here’s a snippet of what I’m dealing with:

async def fetch_block_data():
    data_stream, message_queue = create_data_stream(settings)
    
    while True:
        start = time.time()
        print(f'Listening started at: {start}')
        
        block_data = await message_queue.get()
        
        end = time.time()
        print(f'Data fetch took: {end - start} seconds')
        
        print(f'Block height: {block_data.block.header.height}, Shard count: {len(block_data.shards)}')
        
        await process_block_data(block_data)

# Output example:
# Listening started at: 1660711985
# Data fetch took: 282 seconds
# Block height: 71516637, Shard count: 4

The message_queue.get() part is where it gets stuck for ages. Any ideas what could be causing this? Thanks!

2

hey hugo, i’ve seen similar issues. might be network probs or high traffic. try upping your timeout settings or using a diff node. also, check ur internet connection. sometimes clearing the cache helps too. good luck!

3

hey there hugo! i’ve been playing around with the near lake nft indexer too and ran into some slowdowns. have you considered using a local cache to speed things up? it could help reduce those long waits.

also, what kind of hardware are you running this on? sometimes beefing up your machine can make a big difference.

oh, and here’s a random thought - have you tried running your indexer at different times of day? i noticed it tends to be zippier during off-peak hours. maybe give that a shot?

btw, how many nfts are you trying to index at once? curious if scaling back might help. let me know if any of this helps or if you figure out a solution!