Bulk Data Sync

Sync card data, sets, and prices to your local database for fast queries and offline access.

When to Use Bulk Sync

Building a local search engine
Caching data for faster queries
Offline-first applications
Analytics and reporting

Available Bulk Endpoints

Endpoint	Description	Size
`/v1/bulk/cards`	All card data	~500MB
`/v1/bulk/sets`	All set data	~5MB
`/v1/bulk/prices`	Current prices	~50MB
`/v1/bulk/prices/history`	Historical prices	~2GB

Initial Sync

Download bulk data

import { PullsAPI } from '@pulls/sdk'
import { createWriteStream } from 'fs'

const pulls = new PullsAPI({ apiKey: process.env.PULLS_API_KEY! })

// Stream cards to file
const cardStream = await pulls.bulk.cards()
const fileStream = createWriteStream('cards.jsonl')

for await (const chunk of cardStream) {
  fileStream.write(JSON.stringify(chunk) + '\n')
}

fileStream.close()
console.log('Cards synced')

Import to database

import { createReadStream } from 'fs'
import { createInterface } from 'readline'
import { db } from './database'

const rl = createInterface({
  input: createReadStream('cards.jsonl')
})

for await (const line of rl) {
  const card = JSON.parse(line)
  await db.cards.upsert({
    where: { id: card.id },
    update: card,
    create: card,
  })
}

Record sync timestamp

await db.syncMeta.upsert({
  where: { key: 'lastCardSync' },
  update: { value: new Date().toISOString() },
  create: { key: 'lastCardSync', value: new Date().toISOString() },
})

Incremental Updates

After initial sync, use delta endpoints to fetch only changes:

async function incrementalSync() {
  const lastSync = await db.syncMeta.findUnique({
    where: { key: 'lastCardSync' }
  })

  const updates = await pulls.bulk.cards({
    updatedSince: lastSync?.value ?? new Date(0).toISOString()
  })

  for await (const card of updates) {
    await db.cards.upsert({
      where: { id: card.id },
      update: card,
      create: card,
    })
  }

  await db.syncMeta.upsert({
    where: { key: 'lastCardSync' },
    update: { value: new Date().toISOString() },
    create: { key: 'lastCardSync', value: new Date().toISOString() },
  })
}

// Run daily
setInterval(incrementalSync, 24 * 60 * 60 * 1000)

Price Sync Strategy

Prices change frequently. Sync strategy depends on your needs:

Real-time (Pro+)

Use webhooks for instant price updates:

// Configure webhook at pulls.app/settings/webhooks
// Endpoint receives POST with price changes
app.post('/webhooks/prices', async (req, res) => {
  const { cardId, newPrice, oldPrice, change } = req.body
  await db.prices.update({
    where: { cardId },
    data: { market: newPrice }
  })
  res.status(200).send('OK')
})

Periodic (any plan)

Poll the prices endpoint:

async function syncPrices() {
  let cursor: string | undefined

  do {
    const { data, nextCursor } = await pulls.prices.list({
      limit: 1000,
      cursor
    })

    await db.prices.upsertMany(data)
    cursor = nextCursor

  } while (cursor)
}

// Every 15 minutes
setInterval(syncPrices, 15 * 60 * 1000)

Data Format

Bulk endpoints return JSONL (JSON Lines) format:

{"id":"sv7-001","name":"Bulbasaur","tcg":"pokemon",...}
{"id":"sv7-002","name":"Ivysaur","tcg":"pokemon",...}
{"id":"sv7-003","name":"Venusaur","tcg":"pokemon",...}

Each line is a complete JSON object. This format is:

Streamable (no need to load entire file)
Easy to parse line-by-line
Compatible with tools like jq

Storage Estimates

TCG	Cards	Sets	Prices	History (1yr)
Pokemon	~25,000	~350	~50MB	~500MB
MTG	~80,000	~500	~150MB	~2GB
Yu-Gi-Oh!	~12,000	~300	~25MB	~250MB
Lorcana	~1,500	~10	~3MB	~30MB
One Piece	~2,000	~20	~4MB	~40MB

Plan for ~3GB total if syncing all TCGs with 1 year of price history.

Best Practices

Stream, don’t buffer - Process data line-by-line
Use transactions - Batch database writes
Compress storage - Enable gzip for backups
Verify checksums - Bulk responses include checksums
Schedule off-peak - Run full syncs during low-traffic hours