Jobs API

Track the status of background jobs for crawl and discovery operations.

When you trigger a crawl or discovery, Diffy queues the work as a background job. Use these endpoints to monitor job progress and view results.

List Jobs

GET /api/jobs

Query Parameters

Parameter	Type	Description
limit	integer	Max results (default: 20, max: 100)
status	string	Filter by status: waiting, active, completed, failed

Response

{
  "data": [
    {
      "id": "12345",
      "name": "crawl.page",
      "type": "crawl",
      "url": "https://competitor.com/pricing",
      "domainId": "550e8400-e29b-41d4-a716-446655440000",
      "pageId": "550e8400-e29b-41d4-a716-446655440001",
      "state": "completed",
      "progress": 100,
      "attemptsMade": 1,
      "createdAt": "2024-01-15T10:30:00Z",
      "processedAt": "2024-01-15T10:30:05Z",
      "finishedAt": "2024-01-15T10:30:12Z",
      "failedReason": null
    }
  ],
  "meta": {
    "total": 1,
    "page": 1,
    "pageSize": 20
  }
}

Get Job

GET /api/jobs/:id

Response

{
  "data": {
    "id": "12345",
    "name": "discover.pages",
    "type": "discover",
    "url": null,
    "domainId": "550e8400-e29b-41d4-a716-446655440000",
    "pageId": null,
    "state": "completed",
    "progress": 100,
    "attemptsMade": 1,
    "maxAttempts": 3,
    "createdAt": "2024-01-15T10:30:00Z",
    "processedAt": "2024-01-15T10:30:05Z",
    "finishedAt": "2024-01-15T10:30:45Z",
    "failedReason": null,
    "returnValue": {
      "pagesDiscovered": 8,
      "pagesAdded": 8
    }
  }
}

Queue Statistics

GET /api/jobs/stats

Get aggregate statistics for the job queue.

Response

{
  "data": {
    "waiting": 5,
    "active": 2,
    "completed": 150,
    "failed": 3,
    "delayed": 0,
    "total": 7
  }
}

Job States

State	Description
waiting	Job is queued and waiting to be processed
active	Job is currently being processed
completed	Job finished successfully
failed	Job failed after all retry attempts
delayed	Job is scheduled for future execution

Job Types

Type	Description
crawl.page	Crawl a single page for changes
discover.pages	Discover pages on a domain

Error Codes

Code	Description
NOT_FOUND	Job not found or not owned
UNAUTHORIZED	Invalid or missing API key