Quickstart

Create a scraper key, make your first request, and fetch multiple pages.

The Scraper API is plain HTTP. You do not need an SDK.

1. Create a scraper key

   Open the Scraper dashboard, choose API Keys, and click Create key.

   Copy the key immediately. The full value is shown only once; the dashboard keeps only a truncated preview. Scraper keys begin with scrape_live_.

2. Store the key on your server

   Keep the key in a secret manager or server-side environment variable. Do not put it in browser code, a mobile app, or a repository.

   export SIDESHIFT_SCRAPER_KEY="scrape_live_YOUR_KEY"
   

   Scraper keys are separate from SideShift Platform and Connect credentials. Other SideShift key types are rejected on scraper routes.

3. Request a profile

   Send a POST request with JSON and the x-api-key header.

   cURL

   curl -X POST https://app.sideshift.app/api/v1/scrape/tiktok/profile \
     -H "x-api-key: $SIDESHIFT_SCRAPER_KEY" \
     -H "Content-Type: application/json" \
     -d '{ "username": "mrbeast" }'
   
4. Read the envelope

   The resource payload is in data. Request tracing and billing metadata are always outside it.

   {
     "data": {
       "username": "mrbeast",
       "display_name": "MrBeast",
       "follower_count": 120000000
     },
     "request_id": "req_8f3c9a2b1d4e6f70",
     "upstream_calls": 1,
     "meta": {
       "credits_charged": 1,
       "credits_remaining": 9999
     }
   }
   

   Log request_id, meta.credits_charged, and meta.credits_remaining for every request. Standard profile, post, posts-page, and platform-operation lookups cost one credit; TikTok audience costs 25. If your account has negotiated rates, the dashboard and meta.credits_charged show the exact amount.

Fetch a page of posts

Change the resource to posts. The normalized listing response contains posts, profile_pictures, and next_cursor when another page is available.

curl -X POST https://app.sideshift.app/api/v1/scrape/instagram/posts \
  -H "x-api-key: $SIDESHIFT_SCRAPER_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "username": "nasa", "include_carousels": true }'

Pass a returned next_cursor back as cursor. Never parse, edit, or construct a cursor.

const endpoint = "https://app.sideshift.app/api/v1/scrape/tiktok/posts";
const posts = [];
let cursor;

do {
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      "x-api-key": process.env.SIDESHIFT_SCRAPER_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ username: "mrbeast", ...(cursor ? { cursor } : {}) }),
  });

  const result = await response.json();
  if (!response.ok) throw new Error(`${result.error}: ${result.message}`);

  posts.push(...result.data.posts);
  cursor = result.data.next_cursor || undefined;
} while (cursor);

Each page is one separately billed request at your account’s posts rate (one credit at the standard rate). Snapchat and LinkedIn listings are single-page only and never return a cursor.

Production checklist

• Use a background job for scraping rather than blocking an interactive page request.
• Set a request timeout of at least 60 seconds; use 90 seconds for YouTube and Facebook.
• Store platform post id plus platform as your deduplication key.
• Download short-lived media URLs instead of saving the URL.
• Retry only transient, refunded failures and honor Retry-After on 429 responses.

Understand responses

Learn the normalized and platform-specific data contracts.

Add safe retries

Copy the retry wrapper and review billing rules before going live.