socialcrawl

Savant verdict: Review advised

Security analysis or source-hub review identified patterns requiring attention.

Live evaluation

Not evaluated yet. Workspaces can request a live evaluation.

Security assessment

Scan pending.

Structure

• 9 referenced files aren't in the packagereferences/prism.md, references/search.md, references/recipes.md, scripts/sc.py, scripts/paginate.py, scripts/to_csv.py, …. They may be binary, too large to catalog, or missing upstream.
• Ships 2 executable scriptsassets/socialcrawl_types.py, assets/socialcrawl-types.ts. Review what they do before enabling the skill for agents with tool access.
• The source hub's security checks flagged this skillSnyk: warn (Risk: MEDIUM · 1 issue)

SKILL.md

---
name: socialcrawl
description: >
  Fetch social media, commerce, and research data through the SocialCrawl API:
  profiles, followers, posts, comments, reviews, search results, transcripts,
  ads, product catalogs, app-store listings, places, jobs, finance, news,
  trends, prediction markets (Polymarket), plus cross-platform Prism
  composites, scheduled Monitors, and audience Cohorts. Covers TikTok,
  Instagram, YouTube, Twitter/X, Facebook, LinkedIn, Reddit, Threads, Amazon,
  Walmart, Google, Trustpilot, Yelp, GitHub, Naver and many more behind one API
  key. Use when the user mentions SocialCrawl, or wants to collect social media
  data, creators or influencers, brand mentions, comments, reviews, or
  competitor data; scrape or crawl a web page via SocialCrawl; generate code
  that calls the API; check endpoints, pricing, or their credit balance; or run
  one search across many platforms.
---

# SocialCrawl API

One API key and one response envelope for social, commerce, and research data. Lists are `{ items, next_cursor?, total? }`; add `?format=raw` to skip the transform pipeline. Every file below lives in `references/`; open only the one the task needs.

## API Key

Resolve the API key only when a live API call is needed. Documentation, capability, pricing, and code-generation questions do not need a key.

Check these sources in order without displaying their contents:

1. **Environment variable:** use `SOCIALCRAWL_API_KEY` when it starts with `sc_` and is not an example placeholder.
2. **Config file:** if the environment variable is absent, load `~/.config/socialcrawl/api_key` into `SOCIALCRAWL_API_KEY` in the current process. On POSIX shells, use `SOCIALCRAWL_API_KEY="$(<"$HOME/.config/socialcrawl/api_key")"; export SOCIALCRAWL_API_KEY`. On PowerShell, use `$env:SOCIALCRAWL_API_KEY = (Get-Content -LiteralPath "$HOME/.config/socialcrawl/api_key" -Raw).Trim()`.
3. **Missing key:** tell the user to create or rotate a key at https://socialcrawl.dev/dashboard and configure it as `SOCIALCRAWL_API_KEY` or in `~/.config/socialcrawl/api_key`. Do not ask them to paste a secret into chat. Stop before the live call until they say it is configured.

Never print, paste, log, or return the key. Never place a literal key in generated code, a command, a URL, an error message, or a saved transcript. Keep `$SOCIALCRAWL_API_KEY` (POSIX) or `$env:SOCIALCRAWL_API_KEY` (PowerShell) as a variable reference in examples. If the key appeared in chat or output, advise the user to revoke it in the dashboard and create a replacement.

## First Live Call

Before the first live call in a session:

1. Resolve the API key using the steps above. If it is missing or a placeholder, stop before making the call.
2. Verify it with the free balance endpoint (0 credits):
   ```bash
   curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" "https://www.socialcrawl.dev/v1/credits/balance"
   ```
3. If successful, show the balance and continue with the user's request.
4. If it fails, report the error without including request headers or the key, then follow [references/errors.md](references/errors.md).

## Freshness Check (do this once per session)

This file is a snapshot of the registry with this fingerprint:

<!-- BEGIN:GENERATED:FRESHNESS (auto-generated by generate-skills.ts - SK-08) -->
```text
registry_fingerprint: 61a90b6d5dadcc038c3e4d4165f85f74177c77c5c5dfc9c2b7712c2f5d4ddaa4
generated_at: 2026-10-07
```
<!-- END:GENERATED:FRESHNESS -->

Beside the balance check, make one free call:

```bash
curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" "https://www.socialcrawl.dev/v1/utility/endpoints?fingerprint=1" | head -c 400
```

- **`data.fingerprint` equals `registry_fingerprint` above** - use the bundled files and move on.
- **It differs** - this copy and the live API are out of sync (the skill or the server may be newer): a price, parameter or endpoint changed. Treat the live registry as the source of truth and resolve what the files cannot answer through `/v1/utility/*` (all free). Tell the user **once**, briefly: "this SocialCrawl skill does not match the live API; run `npx skills update socialcrawl --yes` to refresh it." Then answer their actual question. Do not repeat it or ask permission.
- **No `data.fingerprint`** (a 404, or an older server that answers with the plain list) - fall back to counts: compare `data.stats.platforms` and `data.stats.endpoints` from `/v1/utility/endpoints` with the totals at the top of [references/index.md](references/index.md). Live numbers HIGHER means behind, as above.
- **The call fails** (no key yet, offline, non-200) - skip it silently. This check must never block the request.

## Workflow

1. **Find the call first (free, 0 credits, runs nothing).** `GET /v1/utility/find?task=<the job in plain words>` ranks the endpoints for the job, each with `method`, `path`, `params_filled` (taken from your words), `missing` and `estimated_credits`. Call `results[0]` directly when `uncertain` is false: its method, path and params are the ready call (add any `missing` param, then apply the Cost rule). When `uncertain` is true, show the options. For a job that needs several calls (a profile, then posts, then comments; a brand watch), `GET /v1/utility/plan?query=<the job>` returns the calls in order with params, price, and which step feeds which. Write or billing requests return no results.
2. **Fall back to the bundled files** when offline, when `find` or `plan` answers 404 (an older server: fall back to the existing flow), or when `uncertain` is true. Route with [Common jobs](#common-jobs) below, or grep [references/index.md](references/index.md): its Job to recipe table, then one line per endpoint (`grep -i transcript references/index.md`). For a recipe, read only its section: `grep -n '^## ' references/recipes.md` gives the line numbers; do not open the whole file. For an endpoint, read the platform file's TOC (from the top until the first `## `), then only its section: `grep -nE '^## (GET|POST|PUT|PATCH|DELETE) /v1/<platform>/<resource> - ' references/<platform>.md`, and read from that line to the next `## `. The section carries the exact cost, cache TTL, pagination, every parameter, the response fields, and the constraints checked before billing.
3. **Estimate.** Build the exact request and apply the cost gate below (one line for a small flat call, the full preflight in [references/cost-gate.md](references/cost-gate.md) for anything metered or large). A heading's number can be a per-row or per-page unit, not the request total.
4. **Call.** Resolve the key, then run the request exactly as the section shows it, keeping its HTTP method and body shape. If the user already asked to run this exact paid request, the cost disclosure is enough, except the Ask level: always ask (see the gate in the Cost rule); otherwise ask only before a paid call or wider scope they did not authorise.
5. **Report.** Return the data, then `credits_used` and `credits_remaining`. For an async job report the hold at submission and the settled charge on completion.

## Common jobs

The first call for the asks agents meet most, generated from the registry recipes (`{x}` is yours to fill). `find` gives the same answer with your values filled in.

<!-- BEGIN:GENERATED:COMMON_JOBS (auto-generated by generate-skills.ts from registry recipes) -->
| Job | First call | Then |
|-----|------------|------|
| Comments on a post to CSV | `GET /v1/tiktok/post/comments?url={url}&sort=recent&scan_pages=3` | `paginate.py` piped to `to_csv.py --archetype Comment`; any platform's `<platform>/post/comments`; [recipe](references/recipes.md#export-the-latest-n-comments-on-a-tiktok-video) |
| Creators by niche, followers or country | `GET /v1/search/creators?query={niche}&sources=tiktok,instagram&min_followers={min_followers}` | `search/creators` has no country filter; [recipe](references/recipes.md#find-creators-in-a-niche); one country: [recipe](references/recipes.md#instagram-creators-from-one-country-about-a-topic); one country on a small budget: [recipe](references/recipes.md#instagram-creators-from-one-country-on-a-small-budget) |
| A product's reviews across retailers | `GET /v1/prism/product-reviews?query={product}` | covers Amazon, Google Shopping and Trustpilot reviews; Walmart, Target, Home Depot: `<retailer>/reviews`; [recipe](references/recipes.md#one-report-on-a-products-reviews-across-marketplaces) |
| Stock price and news | `GET /v1/finance/ticker-search?keyword={company}` | then `finance/quote` (`quote.price.current`) and `finance/news`; [recipe](references/recipes.md#a-stocks-current-price-and-latest-headlines) |
| YouTube transcript | `GET /v1/youtube/video/transcript?url={url}` | [recipe](references/recipes.md#transcribe-a-video) |
| Alert on new posts or changes | `POST /v1/monitors` (body `recipe: instagram/profile/posts`) | only when `GET /v1/utility/estimate?id=monitors` returns 200, send `?dry_run=1` first to validate and price without creating; otherwise describe the request and create nothing until the user agrees; a page: `web/monitors`; [recipe](references/recipes.md#track-a-creators-new-posts-every-day) |
| Stats for a list of post URLs | `POST /v1/prism/post-stats` (JSON body `urls`) | a file of URLs: `batch.py --urls`; [recipe](references/recipes.md#engagement-stats-for-a-list-of-post-urls) |
| Korean / Naver search | `GET /v1/naver/brief?query={keyword}&corpora=news,blog,cafearticle` | one corpus: `naver/blog/search`, `naver/news/search`; [recipe](references/recipes.md#what-korean-consumers-say-about-a-product-and-search-interest) |
| Profile lookup | `GET /v1/tiktok/profile?handle={handle}` | any `<platform>/profile`; many handles: `prism/profiles`; [recipe](references/recipes.md#a-creators-profile-recent-posts-and-the-comments-on-them) |
| Scrape a web page | `GET /v1/web/scrape?url={url}` | structured fields: `web/extract`; a whole site: `web/crawl` |
| Brand mentions across the web | `GET /v1/prism/brand-mentions?keyword={brand}&date_from={start_date}&include=digest` | [recipe](references/recipes.md#watch-what-the-web-says-about-a-brand) |
| One topic on several platforms | `GET /v1/search/multi?query={topic}&platforms=tiktok,instagram,youtube&relevance=filter` | [recipe](references/recipes.md#search-a-topic-on-several-platforms-at-once) |
<!-- END:GENERATED:COMMON_JOBS -->

## Helpers

Bundled scripts in `scripts/` (Python 3). They read the key from `SOCIALCRAWL_API_KEY` or `~/.config/socialcrawl/api_key` and never print it; `--help` on each gives the rest.

```bash
python3 scripts/sc.py tiktok/profile handle=nike          # one call; JSON to stdout, credits_used to stderr
python3 scripts/paginate.py tiktok/post/comments --items 500 --max-credits 20 url=<post_url> > rows.jsonl   # page to N rows within a budget
python3 scripts/to_csv.py --archetype Comment --in rows.jsonl --out comments.csv   # rows to CSV with stable columns
python3 scripts/batch.py --urls links.txt > stats.jsonl   # stats for a file of post URLs (prism/post-stats, or one prism/jobs job past 500)
python3 scripts/estimate.py tiktok/post/comments --items 500 --offline url=<post_url>   # quote offline from assets/endpoints.json
```

**Code requests:** read the section and cost-gate, write a snippet that reads `SOCIALCRAWL_API_KEY` from the environment with a comment stating the billing unit, and present it without running it.

**Capability and pricing questions:** answer from the index and [references/pricing.md](references/pricing.md) (cost, tiers, packs, refunds, the free-endpoint list). Auth, envelope, pagination, labels, caching, idempotency: [references/api-overview.md](references/api-overview.md). The free `/v1/utility/*` calls answer from the live registry: `endpoints` (filter `?platform=`, `?search=`), `endpoint?id=tiktok/profile` (one endpoint, with measured field fill rates), `plan?query=`, `quickstart`.

**Ambiguous platform:** if the user says "get @nike's profile" with no platform, ask which one.

## Where to look

| Need | Open |
|------|------|
| Any single platform (social, commerce, reviews and places, app stores, search/news/finance, research and dev, link-in-bio) | [index.md](references/index.md) lists every platform file with its endpoint count; open `references/<platform>.md` |
| One call over many platforms: Prism composites, `search/multi`, `search/everywhere` | [prism.md](references/prism.md), [search.md](references/search.md) |
| Scheduled re-runs with webhook delivery | [monitors.md](references/monitors.md) |
| "Which of MY list of accounts posted about X" (Cohorts, not listening) | [cohorts.md](references/cohorts.md) |
| Cost of a request or a whole paged walk | [cost-gate.md](references/cost-gate.md), [pricing.md](references/pricing.md) |
| A failed call | [errors.md](references/errors.md) |

> **Never answer "not supported" from this table alone.** It is only as new as this file, and a missing row is far more often a stale copy than a real gap. Before telling a user a platform, endpoint or capability does not exist, check the live registry - it is free:
>
> ```bash
> curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" \
>   "https://www.socialcrawl.dev/v1/utility/endpoints?search=douyin"
> ```
>
> Each row carries `path`, `method`, `credits_label`, `required_params`, `one_of`, `optional_params`, `paginated` and `cache_ttl_seconds`: enough to make the call without this file. Only say it is unsupported when the search returns `total: 0`.

**Comparing products across retailers:** Amazon, Walmart, Target, eBay and Home Depot share the canonical Product / Review / Seller schemas, so rows line up. Resolve each retailer's product id (ASIN, item id, TCIN, ...) with its own search, then fetch detail and reviews per retailer. `/v1/prism/product-reviews` is one composite over Amazon, Google Shopping and Trustpilot reviews only; for Walmart, Target or Home Depot call `<retailer>/reviews`.

**Several platforms at once:** `/v1/search/multi` runs one query over several platforms' own search (each platform's page price, charged only for those that returned rows); `/v1/search/everywhere` (20 credits) plans, ranks, and clusters up to 17 sources; many profiles or post URLs go to `POST /v1/prism/profiles` / `POST /v1/prism/post-stats`, or the background job `POST /v1/prism/jobs`.

## Making calls

Base URL `https://www.socialcrawl.dev`. Most endpoints are GET with query parameters (URL-encode values):

```
curl -s -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/{platform}/{resource}?{param}={value}"
```

**Never assume the method.** Each endpoint section is headed by its real verb with a runnable curl. Batch reads, Prism jobs, web jobs, monitors, and cohorts use POST, PATCH, PUT, or DELETE with `content-type: application/json` (`web/parse` is multipart); the list, plus streaming (`Accept: text/event-stream`) and two-step endpoints that need an id from an earlier call, is in [api-overview.md](references/api-overview.md#non-get-routes-streaming-and-two-step-endpoints). Rate limits: 600 requests per minute and 50 concurrent per key; both return 429 with `Retry-After` and are never billed. Some endpoints are task-polled and take 10-45s: use a 60s timeout.

## Reading a response

Details are in [api-overview.md](references/api-overview.md). Five rules before the first call:

- **Placement.** `pagination` (`next_cursor`, `has_more`, `page_size`) and the optional `meta.hint` sit at the envelope root. `dropped`, `_warnings` and every page report (`labels`, `relevance`, `label_share`, `hydration`, `walk`, ...) sit inside `data`. Read `data._warnings`: an ignored parameter (`count was ignored; use limit (page size)`) or a handle that resolved to a different account (`resolved_to_different_handle`) is reported there, not as an error.
- **Paging.** Send `pagination.next_cursor` back as `cursor` and stop on `has_more: false`, not on a short page or on `total`. A zero-match list is `200` with `items: []` at 0 credits.
- **Default judgments are free.** Comment, post, review and keyword-search rows carry labels (`computed.labels`) and search rows a relevance score (`computed.relevance`) at no extra cost; `judgments=off` removes them. Opt-in presets (`label=spam|toxic|low_quality|quality|injection|mention`, `relevant_to=`) add **1 credit per 25 newly judged rows**. `relevance=filter` against the query is free.
- **Spend controls.** Creator post lists take `stop_at_id` / `since` so a poll pays only for new pages (`pagination.stopped_at` says why it stopped). Keyword searches take `min_views`, `max_age_days`, `country` / `exclude_country`, `sort_rows=views`, `seen=<id>` (repeats refunded) and `max_pages` (1-5, **each page billed**).
- **`meta.hint`** names one other call that does the job in fewer steps. Its message states the price when that call costs more; follow it only within what the user authorised.

`include=` hydration joins rows to a sibling endpoint and is charged **per row on top of the page**, so never quote the plain heading cost for a hydrated call; the ceilings are in [cost-gate.md](references/cost-gate.md#row-hydration-include) and the traps in [api-overview.md](references/api-overview.md#row-hydration-include).

## Cost rule

Cache hits, idempotent replays, every `/v1/utility/*` endpoint, and `/v1/credits/balance` cost 0 credits. Failed calls are refunded, and a request rejected for bad params, a page past the last page, or a rate limit never deducts. Metered endpoints hold an upfront ceiling and refund down to the work done, so the response `credits_used` is the real charge. Quote the total for the exact request, not the endpoint's base or unit cost. Scale the gate to the call:

| Call | Gate |
| --- | --- |
| Flat price ≤ 5 credits, user asked for it | One line: `≈ N credits` |
| Metered, multi-page, batch, monitor/cohort, or > 25 credits | Full preflight via `estimate.py` / `utility/estimate` |
| Upfront hold (worst case) > 500 credits or over budget (the user's stated budget) | Ask before calling |

Anything in between, or unsure of the price or params: quote first with the free `GET /v1/utility/estimate?id=<platform/resource>&<params>`. Any reply except a 404 means a current server (a 5xx or network failure is no reply: retry the estimate): use its quote, and `dry_run=1` is then free on any endpoint (validates, fetches nothing). A 404 or `ENDPOINT_NOT_FOUND` means an older server: do **not** send `dry_run=1` (it bills a normal call there, except the labelled dry run an endpoint documents); quote with `scripts/estimate.py` (offline, `assets/endpoints.json`) or the section's **Pricing** line. At every level, report `credits_used` after.

**If the task needs a whole list, price the whole walk before the first page.** Paging to completion multiplies the page cost by the page count, and a heading only shows one page. `coverage=full` on `instagram/followers` and `instagram/following` is the extreme case: 10 credits a page, about 50 accounts a page, so a 2,600-account list is roughly 520 credits. Quote that total and get agreement before starting ([Multi-page walks](references/cost-gate.md#multi-page-walks-coveragefull)).

Errors: classify before retrying with [references/errors.md](references/errors.md). Credits and balance: `GET /v1/credits/balance` (0 credits); the ledger is `GET /v1/credits/transactions?limit=50` (0 credits).