Introduction

Anyone who’s worked with NFT data knows that keeping everything in sync and up to date is a challenge. Very little is static when it comes to NFTs. As a few (non-exhaustive) examples:

• Token ownership changes frequently, along with mints and burns
• Chain re-orgs can change the transaction history
• New NFT drops or reveals of existing collections can happen at any time
• Dynamic NFTs change their metadata or media
• Collections themselves can update, split, or change their metadata, and collections can be delisted
• Sales occur across many different marketplaces
• The floor price of a collection can update at any moment, based on the active listings
• The top bids, or bids per token are also always in flux

Some of this data can be found on-chain, depending on the network and NFT standard(s) involved, but much of it involves retrieving off-chain data too. When considering the number of tokens and collections in existence (billions and millions respectively), it becomes clear that dealing with all of this is a large real-time data, storage and computation challenge.

Without a proper indexing solution, any aspect of token or collection level information can quickly become stale or incorrect, leading to a poor end-user experience, or create unacceptable errors for trading or accounting applications.

In this blog post, we’re going to break down each of the ways that SimpleHash works to keep all of these factors in sync in order to provide the most reliable and up to date NFT data.

Keeping token ownership accurate

Correctly identifying the ownership of an NFT is at the heart of what makes them non-fungible (or semi-fungible). SimpleHash listens to all log / transaction events that occur on-chain across more than 40 different blockchains. For every transaction that we identify as a token transfer, we reference whether this is a new NFT or not, and then update the cached ownership in our system accordingly. We correctly handle situations where tokens have a single owner (e.g., ERC721s on EVM-based chains), vs. multiple owners (e.g., ERC1155 tokens). Typically, ownership changes propagate in less than 5 seconds before being reflected everywhere.

Dealing with chain re-orgs

A chain re-org, or chain reorganization, refers to the event in blockchain networks when one chain becomes longer than the one currently being used, leading to the replacement of the shorter chain. This can happen in decentralized networks due to the nature of their distributed consensus mechanism. In the context of NFTs and other blockchain-based operations, a chain re-org can have significant implications. Transactions that were confirmed in the now-discarded blocks of the shorter chain can become unconfirmed. If, for instance, an NFT was sold in one of those discarded blocks, that sale could be effectively "erased," and the NFT might return to its previous owner.

SimpleHash’s transfer table is optimized for ingestion speed, so we provide ownership updates as they occur. If subsequently there is a chain re-org however, we will automatically update the relevant transaction in question.

Developers can keep track of when chain re-orgs occur by using our Kafka Bulk Service, which will send a DELETE message when a transaction that is no longer valid due to a re-org occurs. An alternative is for consumers to wait 10s to 60s if absolute finality is always required in their application, or to by checking the Sales & Transfers REST API endpoints on an ongoing basis.

‍

Covering drops, reveals and dynamic NFTs

It’s common for new NFT collections to not feature their final media or metadata until some time after their mint - or even to change on an ongoing basis in the case of dynamic NFTs. Not catching these changes will result in the information regarding a specific token to be stale, or incorrect.

SimpleHash uses multiple approaches to continually keep everything fresh and inline with how collections and tokens evolve:

• On-chain metadata events: Each ecosystem has a different set of standards for implementing the emission of events for when NFT metadata and media changes. On EVM that is often the ERC-4906 standard. On Solana, that involves updates to the Metaplex Token Metadata Account. Each time the SimpleHash indexer sees these events, it will trigger a refresh for the collection in question, after running through a series of other checks.‍
• Marketplace refreshes: Many NFT marketplaces allow users to refresh collections on demand. SimpleHash listens to the flow of these signals, and triggers updates of collections if they meet a set of relevant criteria.‍
• Smart heuristics: The SimpleHash indexer is also watching the media and metadata coverage / similarity ratios of collections, especially for dynamic collections. When ratios fall below a certain threshold, new refreshes are triggered, using an exponential backoff if the new media / metadata is unavailable or times out ‍
• Manual refreshes: if required, users can also request manual refreshes of tokens, wallets, collections or contracts.

SimpleHash also makes it really easy to track these metadata or media updates, either by using our webhook events for this purpose, or our Kafka stream of token metadata.

Performing collection updates

It’s not just token level data that can change. Collections themselves can change, both the metadata (e.g., name, description), but also the holdings of a given collection. Depending on the actions of the collection creator, collections may subsequently be updated, split or merged. Since collections can be defined in many different ways depending on the chain or standard (e.g., at the marketplace level, on-chain, or otherwise), this can be a headache to keep up with.

SimpleHash also abstracts away this complexity by keeping collections up to date. The indexer is continually sampling collections and comparing tokens against the known on-chain state, or on the relevant marketplace, depending on the collection type. When changes are reflected, the collection ownership will be updated for all relevant tokens.

There are other key events for collections - they can be delisted from marketplaces for various reasons (e.g., copyright issues, fraud, spam, or otherwise). Knowing when this occurs is very important for ultimately providing a good end-user NFT experience - SimpleHash uses this listing status as one factor in determining collection level spam scoring. Accordingly, we’re continually scanning marketplace activity, and collection metadata on marketplaces to ensure listing / delisting status is up to date.

Providing sales, floor price, listing and bid changes

Keeping the on-chain information, metadata and media up to data is only half of the equation. Since many NFTs and collections are traded actively on marketplaces, keeping all sales, floor prices, listings and bids up to date is also crucial.

SimpleHash integrates with all of the major marketplaces on each ecosystem (including the marketplaces that cover more than 99% of EVM volume), and ingests each core market data type.

Sales may be conducted on chain, or off chain depending on the marketplace. Our indexer unifies the schema for each sale contract, and for off-chain sales, tracking each event as new logs are ingested, or through polling for updates via marketplace APIs.

Floor prices are intimately linked to listings. The lowest active listing for a given collection usually drives the floor price, so having an accurate and up to date state is crucial. Again, SimpleHash ingests listing data from various marketplace sources, and then updates the relevant collection or trait-level floor price accordingly.

Bids are a similar story - each ingested bid, or top active collection bid is ingested, and then applied to the relevant collection.

For heavily traded collections, listings and bids can change many times per second - our indexing architecture is designed to keep up with all of this, and to ensure that everything stays up to date.

‍

Conclusion

In the fast-paced realm of NFTs, data accuracy and real-time updates are non-negotiable. Tracking token ownership, understanding chain re-orgs, adjusting to dynamic NFT alterations, and monitoring ever-evolving marketplace dynamics are complex tasks. For developers and product leaders, building and continuously maintaining an in-house solution to handle these intricacies isn't just resource-intensive, it can quickly become a costly distraction from core objectives.

SimpleHash offers a pragmatic alternative. Instead of diving deep into the trenches of NFT data management, developers can integrate with a platform that's been fine-tuned to manage the nuances. It's about optimizing your time, reducing overheads, and maintaining an agile operation. For leaders in the NFT and crypto space, choosing the right NFT data provider isn't just about outsourcing a task; it's about strategically positioning your startup for growth and agility in a competitive domain.

‍