DataForSEO
SEO, search-engine results, keyword research, backlinks and AI visibility data.
OnPage API Non-indexable Pages
The crawled pages search engines will not index, each with a reason and url. Reads a finished crawl, so it needs the id from post_dataforseo_on_page_submit and returns crawl_progress plus a crawl_status of max_crawl_pages, pages_in_queue and pages_crawled - check those before trusting a small result, because a crawl still running simply has less to report. The reason field is the whole value here - noindex, canonical elsewhere, robots-blocked are very different problems with the same symptom. 💰 Free upstream: querying a finished crawl costs nothing, only the crawl itself does.
- Type: array object[]
- idType: stringrequired
ID of the task required field you can get this ID in the response of the Task POST endpoint example: “07131248-1535-0216-1000-17384017ad04”
- filtersType: array
array of results filtering parameters optional field you can add several filters at once (8 filters maximum) you should set a logical operator and, or between the conditions the following operators are supported: regex, not_regex, , , >, >=, =, , in, not_in, like, not_like you can use the % operator with like and not_like to match any string of zero or more characters example: ["reason","=","robots_txt"][["reason","","robots_txt"], "and", ["url","not_like","%/wp-admin/%"]] [["url","not_like","%/wp-admin/%"], "and", [["reason","","meta_tag"],"or",["reason","","http_header"]]] The full list of possible filters is available by this link.
- limitType: integer
the maximum number of returned pages optional field default value: 100 maximum value: 1000
- offsetType: integer
offset in the results array of returned pages optional field default value: 0 if you specify the 10 value, the first ten pages in the results array will be omitted and the data will be provided for the successive pages
- Type: object200
Successful response
- Type: numbercost
total tasks cost, USD
- Type: integerstatus
_code general status code you can find the full list of the response codes here Note: we strongly recommend designing a necessary system for handling related exceptional or error conditions
- Type: stringstatus
_message general informational message you can find the full list of general informational messages here
- Type: array string[]tasks
array of tasks
- Type: integertasks
_count the number of tasks in the tasks array
- Type: integertasks
_error the number of tasks in the tasks array returned with an error
- Type: numbertasks
.cost cost of the task, USD
- Type: objecttasks
.data contains the same parameters that you specified in the POST request
- Type: stringtasks
.id task identifier unique task identifier in our system in the UUID format
- Type: array string[]tasks
.path URL path
- Type: array string[]tasks
.result array of results
- Type: integertasks
.result _count number of elements in the result array
- Type: stringtasks
.result .crawl _progress status of the crawling session possible values: in_progress, finished
- Type: objecttasks
.result .crawl _status details of the crawling session
- Type: integertasks
.result .crawl _status .max _crawl _pages maximum number of pages to crawl indicates the max_crawl_pages limit you specified when setting a task
- Type: integertasks
.result .crawl _status .pages _crawled number of crawled pages
- Type: integertasks
.result .crawl _status .pages _in _queue number of pages that are currently in the crawling queue
- Type: array string[]tasks
.result .items items array
- Type: integertasks
.result .items _count number of items in the results array
- Type: stringtasks
.result .reason the reason why the page is non-indexable can take the following values: robots_txt, meta_tag, http_header, attribute, too_many_redirects
- Type: integertasks
.result .total _items _count total number of relevant items in the database
- Type: stringtasks
.result .url url of the non-indexable page
- Type: integertasks
.status _code status code of the task generated by DataForSEO; can be within the following range: 10000-60000 you can find the full list of the response codes here
- Type: stringtasks
.status _message informational message of the task you can find the full list of general informational messages here
- Type: stringtasks
.time execution time, seconds
- Type: stringtime
execution time, seconds
- Type: stringversion
the current version of the API
application/json - 400
Bad request
- 401
Unauthorized
- 402
The request cannot proceed because a billing requirement is not met.
- 403
The account is not permitted to perform this operation.
- 429
Rate limit exceeded
- 500
Internal server error
- 502
The service could not complete the request.
- 503
The API or billing service is temporarily unavailable.
- 504
The service timed out while processing the request.
- default
The operation failed. Keep the response request ID when contacting Felo support.
curl https://openapi.felo.ai/v1/beta/dataforseo/on_page/non_indexable \
--request POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_SECRET_TOKEN' \
--data '[
{
"id": "",
"limit": 1,
"offset": 1,
"filters": []
}
]'
{
"version": "string",
"status_code": 1,
"status_message": "string",
"time": "string",
"cost": 1,
"tasks_count": 1,
"tasks_error": 1,
"tasks": [
"string"
],
"tasks.id": "string",
"tasks.status_code": 1,
"tasks.status_message": "string",
"tasks.time": "string",
"tasks.cost": 1,
"tasks.result_count": 1,
"tasks.path": [
"string"
],
"tasks.data": {},
"tasks.result": [
"string"
],
"tasks.result.crawl_progress": "string",
"tasks.result.crawl_status": {},
"tasks.result.crawl_status.max_crawl_pages": 1,
"tasks.result.crawl_status.pages_in_queue": 1,
"tasks.result.crawl_status.pages_crawled": 1,
"tasks.result.total_items_count": 1,
"tasks.result.items_count": 1,
"tasks.result.items": [
"string"
],
"tasks.result.reason": "string",
"tasks.result.url": "string"
}