Anyone who’s worked with NFT data knows that keeping everything in sync and up to date is a challenge. Very little is static when it comes to NFTs. As a few (non-exhaustive) examples:
Some of this data can be found on-chain, depending on the network and NFT standard(s) involved, but much of it involves retrieving off-chain data too. When considering the number of tokens and collections in existence (billions and millions respectively), it becomes clear that dealing with all of this is a large real-time data, storage and computation challenge.
Without a proper indexing solution, any aspect of token or collection level information can quickly become stale or incorrect, leading to a poor end-user experience, or create unacceptable errors for trading or accounting applications.
In this blog post, we’re going to break down each of the ways that SimpleHash works to keep all of these factors in sync in order to provide the most reliable and up to date NFT data.
Correctly identifying the ownership of an NFT is at the heart of what makes them non-fungible (or semi-fungible). SimpleHash listens to all log / transaction events that occur on-chain across more than 40 different blockchains. For every transaction that we identify as a token transfer, we reference whether this is a new NFT or not, and then update the cached ownership in our system accordingly. We correctly handle situations where tokens have a single owner (e.g., ERC721s on EVM-based chains), vs. multiple owners (e.g., ERC1155 tokens). Typically, ownership changes propagate in less than 5 seconds before being reflected everywhere.
A chain re-org, or chain reorganization, refers to the event in blockchain networks when one chain becomes longer than the one currently being used, leading to the replacement of the shorter chain. This can happen in decentralized networks due to the nature of their distributed consensus mechanism. In the context of NFTs and other blockchain-based operations, a chain re-org can have significant implications. Transactions that were confirmed in the now-discarded blocks of the shorter chain can become unconfirmed. If, for instance, an NFT was sold in one of those discarded blocks, that sale could be effectively "erased," and the NFT might return to its previous owner.
SimpleHash’s transfer table is optimized for ingestion speed, so we provide ownership updates as they occur. If subsequently there is a chain re-org however, we will automatically update the relevant transaction in question.
Developers can keep track of when chain re-orgs occur by using our Kafka Bulk Service, which will send a DELETE message when a transaction that is no longer valid due to a re-org occurs. An alternative is for consumers to wait 10s to 60s if absolute finality is always required in their application, or to by checking the Sales & Transfers REST API endpoints on an ongoing basis.
It’s common for new NFT collections to not feature their final media or metadata until some time after their mint - or even to change on an ongoing basis in the case of dynamic NFTs. Not catching these changes will result in the information regarding a specific token to be stale, or incorrect.
SimpleHash uses multiple approaches to continually keep everything fresh and inline with how collections and tokens evolve:
SimpleHash also makes it really easy to track these metadata or media updates, either by using our webhook events for this purpose, or our Kafka stream of token metadata.
It’s not just token level data that can change. Collections themselves can change, both the metadata (e.g., name, description), but also the holdings of a given collection. Depending on the actions of the collection creator, collections may subsequently be updated, split or merged. Since collections can be defined in many different ways depending on the chain or standard (e.g., at the marketplace level, on-chain, or otherwise), this can be a headache to keep up with.
SimpleHash also abstracts away this complexity by keeping collections up to date. The indexer is continually sampling collections and comparing tokens against the known on-chain state, or on the relevant marketplace, depending on the collection type. When changes are reflected, the collection ownership will be updated for all relevant tokens.
There are other key events for collections - they can be delisted from marketplaces for various reasons (e.g., copyright issues, fraud, spam, or otherwise). Knowing when this occurs is very important for ultimately providing a good end-user NFT experience - SimpleHash uses this listing status as one factor in determining collection level spam scoring. Accordingly, we’re continually scanning marketplace activity, and collection metadata on marketplaces to ensure listing / delisting status is up to date.
Keeping the on-chain information, metadata and media up to data is only half of the equation. Since many NFTs and collections are traded actively on marketplaces, keeping all sales, floor prices, listings and bids up to date is also crucial.
SimpleHash integrates with all of the major marketplaces on each ecosystem (including the marketplaces that cover more than 99% of EVM volume), and ingests each core market data type.
Sales may be conducted on chain, or off chain depending on the marketplace. Our indexer unifies the schema for each sale contract, and for off-chain sales, tracking each event as new logs are ingested, or through polling for updates via marketplace APIs.
Floor prices are intimately linked to listings. The lowest active listing for a given collection usually drives the floor price, so having an accurate and up to date state is crucial. Again, SimpleHash ingests listing data from various marketplace sources, and then updates the relevant collection or trait-level floor price accordingly.
Bids are a similar story - each ingested bid, or top active collection bid is ingested, and then applied to the relevant collection.
For heavily traded collections, listings and bids can change many times per second - our indexing architecture is designed to keep up with all of this, and to ensure that everything stays up to date.
In the fast-paced realm of NFTs, data accuracy and real-time updates are non-negotiable. Tracking token ownership, understanding chain re-orgs, adjusting to dynamic NFT alterations, and monitoring ever-evolving marketplace dynamics are complex tasks. For developers and product leaders, building and continuously maintaining an in-house solution to handle these intricacies isn't just resource-intensive, it can quickly become a costly distraction from core objectives.
SimpleHash offers a pragmatic alternative. Instead of diving deep into the trenches of NFT data management, developers can integrate with a platform that's been fine-tuned to manage the nuances. It's about optimizing your time, reducing overheads, and maintaining an agile operation. For leaders in the NFT and crypto space, choosing the right NFT data provider isn't just about outsourcing a task; it's about strategically positioning your startup for growth and agility in a competitive domain.
