Skip to content

feat(retriever): add SubgraphRetriever #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 7, 2025
Merged

Conversation

silent-cipher
Copy link
Collaborator

@silent-cipher silent-cipher commented Jun 23, 2025

This PR introduces the SubgraphRetriever, a new component that allows fetching piece information from a user-configured subgraph.

It can be chained with other retrievers, such as ChainRetriever and FilCDNRetriever, to create a flexible and powerful data discovery pipeline. This provides an alternative for users who rely on subgraphs for commp to provider mapping.

This also contains an SubgraphService that can be used independently -

import { SubgraphService } from '@filoz/synapse-sdk'

const subgraphUrl = 'https://api.thegraph.com/subgraphs/name/some/subgraph'
const subgraphService = new SubgraphService({ endpoint: subgraphUrl })

// Example: Query for active providers with custom filtering
const activeProviders = await subgraphService.queryProviders({
  where: {
    status: 'Approved',
  },
  orderBy: 'totalProofSets',
  orderDirection: 'desc',
  first: 10
})

console.log('Top 10 active providers by proof sets:', activeProviders)

Copilot

This comment was marked as outdated.

@silent-cipher silent-cipher requested a review from rvagg June 23, 2025 11:11
@rjan90 rjan90 moved this to 🐱 Todo in FS Jun 23, 2025
@rjan90 rjan90 added this to the M2: Pandora Alpha on Mainnet milestone Jun 23, 2025
@rvagg
Copy link
Collaborator

rvagg commented Jun 24, 2025

Looking good!

@rjan90 rjan90 moved this from 🐱 Todo to 🔎 Awaiting review in FS Jun 24, 2025
@silent-cipher silent-cipher requested a review from rvagg June 26, 2025 20:06
@rvagg rvagg requested a review from Copilot July 3, 2025 06:35
Copilot

This comment was marked as outdated.

* Returns an empty array if no providers are found or if an error occurs during the fetch.
*/
async getProvidersForCommP (commP: CommP): Promise<ApprovedProviderInfo[]> {
const hexCommP = toHex(commP.bytes)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const hexCommP = toHex(commP.bytes)
const commPParsed = asCommP(commP)
if (commPParsed == null) {
throw new Error(`Invalid CommP: ${commP}`)
}
const hexCommP = toHex(commPParsed.bytes)

@silent-cipher silent-cipher requested a review from Copilot July 5, 2025 12:26
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces subgraph-based piece retrieval by adding a SubgraphService and a SubgraphRetriever, integrates them into the main Synapse flow, and updates types, exports, and documentation accordingly.

  • Extend SynapseOptions with subgraphService and subgraphConfig to configure subgraph retrieval.
  • Implement SubgraphService (GraphQL client) and SubgraphRetriever (with fallback to chain/CDN retrievers).
  • Integrate the new retriever into synapse.ts, export it in the retriever module, and add usage documentation in README.md.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/types.ts Added SubgraphConfig and SubgraphRetrievalService options
src/subgraph/service.ts New GraphQL-based SubgraphService implementation
src/retriever/subgraph.ts New SubgraphRetriever with child fallback
src/synapse.ts Hooked up subgraph retriever in Synapse initialization
src/retriever/index.ts Exported SubgraphRetriever
README.md Added "Subgraph Service" section and updated Synapse options docs
Comments suppressed due to low confidence (1)

src/subgraph/service.ts:336

  • The log prefix 'SubgraphService: queryProviders' in safeConvertHexToCid is misleading since this is CID-conversion logic, not part of queryProviders. Consider updating it to reference safeConvertHexToCid (or removing the incorrect method name) for clarity.
      )

Copy link
Collaborator

@rvagg rvagg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work

@github-project-automation github-project-automation bot moved this from 🔎 Awaiting review to ✔️ Approved by reviewer in FS Jul 7, 2025
@rvagg rvagg merged commit 6352278 into master Jul 7, 2025
14 checks passed
@rvagg rvagg deleted the feat/subgraph-retriever branch July 7, 2025 05:15
@github-project-automation github-project-automation bot moved this from ✔️ Approved by reviewer to 🎉 Done in FS Jul 7, 2025
github-actions bot pushed a commit that referenced this pull request Jul 11, 2025
## [0.19.0](v0.18.0...v0.19.0) (2025-07-11)

### Features

* implement SynapseStorage[#piece](https://github.com/FilOzone/synapse-sdk/issues/piece)Status(commp) ([#127](#127)) ([9ee7f5b](9ee7f5b))
* **retriever:** add SubgraphRetriever ([#115](#115)) ([6352278](6352278))

### Trivial Changes

* Update .github/workflows/add-issues-and-prs-to-fs-project-board.yml [skip ci] ([579ef07](579ef07))
Copy link

🎉 This PR is included in version 0.19.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: 🎉 Done
Development

Successfully merging this pull request may close these issues.

3 participants