Smart Article Extractor

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from oth

liveasyncunknown~30s typical

newsaiarticlesmart

$0.0020

per result + $0.01 call

plus $0.001 platform fee per run

Free actor: you pay compute at $0.002 per result plus $0.01 per run

Bring your own Apify Store key: provider portion drops to $0.

Provider Upstream docs

$Set up https://cracked-ai-pearl.vercel.app/SKILL.md, then use Cracked to run apify /lukaskrivka/article-extractor-smart for me.

Input

Field	Type	Description
startUrls	array	These could be the main page URL or any category/subpage URL, e.g. https://www.bbc.com/. Article pages are detected and crawled from these. If you prefer to use direct article URLs, use `articleUrls` input instead
articleUrls	array	These are direct URLs for the articles to be extracted, e.g. https://www.bbc.com/news/uk-62836057. No extra pages are crawled from article pages.
onlyNewArticles	boolean	This option is only viable for smaller runs. If you plan to use this on a large scale, use the 'Only new articles (saved per domain)' option below instead. If this function is selected, the extractor will only scrape new articles each time you run it. (Scraped URLs are saved in a dataset named `articles-state`, and are compared with new ones.) default false
onlyNewArticlesPerDomain	boolean	If this function is selected, the extractor will only scrape new articles each time you run it. (Scraped articles are saved in one dataset, named 'ARTICLES-SCRAPED-domain', per each domain, and compared with new ones.) default false
onlyInsideArticles	boolean	If this function is selected, the extractor will only scrape articles that are on the domain from where they are linked. If the domain presents links to articles on different domains, those articles will not be scraped, e.g. https://www.bbc.com/ vs. https://www.bbc.co.uk/. default true
enqueueFromArticles	boolean	Normally, the scraper only extracts articles from category pages. This option allows the scraper to also extract articles linked within articles. default false
crawlWholeSubdomain	boolean	Automatically enqueue categories and articles from whole subdomain with the same path. E.g. if Start URL is https://apify.com/store, it will enqueue all pages starting with https://apify.com/store default false
onlySubdomainArticles	boolean	Only loads articles which URL begins with the same path as Start URL. E.g. if Start URL is https://apify.com/store, it will only load articles starting with https://apify.com/store default false
scanSitemaps	boolean	We recommend using `Sitemap URLs` instead. If this function is selected, the extractor will scan different sitemaps from the initial article URL. Keep in mind that this option can lead to the loading of a huge amount of (sometimes old) articles, in which case the time and cost of the scrape will increase. default false
sitemapUrls	array	You can provide selected sitemap URLs that include the articles you need to extract.
saveHtml	boolean	If this function is selected, the scraper will save the full HTML of the article page, but this will make the data less readable.
saveHtmlAsLink	boolean	If this function is selected, the scraper will save the full HTML of the article page as a URL to keep the dataset clean and small.
saveSnapshots	boolean	Stores a screenshot for each article page to Key-Value Store and provides that as screenshotUrl. Useful for debugging. default false
useGoogleBotHeaders	boolean	This option will allow you to bypass protection and paywalls on some websites. Use with caution as it might lead to getting blocked. default false
minWords	integer	The article needs to contain at least this number of words to be extracted default 150
dateFrom	string	Only articles from this day on will be scraped. If empty, all articles will be scraped. Format is YYYY-MM-DD, e.g. 2019-12-31, or number type e.g. 1 week or 20 days
onlyArticlesForLastDays	integer	Only get posts that were published in the last X days from time the scraping starts. Use either this or the absolute date.
mustHaveDate	boolean	If checked, the article must have a date of release to be extracted. default true
isUrlArticleDefinition	object	Here you can input JSON settings to define what URLs should be considered articles by the scraper. If any of them is `true`, then the link will be opened and the article extracted.
pseudoUrls	array	This function can be used to enqueue more pages, i.e. include more links like pagination or categories. This doesn't work for articles, as they are recognized by the recognition system.
linkSelector	string	You can limit the <a> tags whose links will be enqueued. This field is empty by default. Add `a.some-class` to activate it
maxDepth	integer	Maximum depth of crawling, i.e. how many times the scraper picks up a link to other webpages. Level 0 refers to the start URLs, 1 are the first level links, and so on. This is only valid for pseudo URLs
maxPagesPerCrawl	integer	Maximum number of total pages crawled. It includes the home page, pagination pages, invalid articles, and so on. The crawler will stop automatically after reaching this number.
maxArticlesPerCrawl	integer	Maximum number of valid articles scraped. The crawler will stop automatically after reaching this number.
maxArticlesPerStartUrl	integer	Maximum number of articles scraped per start URL.
maxConcurrency	integer	You can limit the speed of the scraper to avoid getting blocked.
proxyConfiguration	object	Proxy configuration
overrideProxyGroup	string	If you want to override the default proxy group, you can specify it here. This is useful if you want to use a different proxy group for each crawler.
useBrowser	boolean	This option is more expensive, but it allows you to evaluate JavaScript and wait for dynamically loaded data. default false
pageWaitMs	integer	How many milliseconds to wait on each page before extracting data
navigationWaitUntil	string (load \| domcontentloaded \| networkidle0 \| networkidle2)	What to wait until the navigation is finished. `domcontentloaded` happens when initial HTML loads and is fastest. `load` happens when JS is executed and it is default. `networkidle0`, `networkidle2` wait for background network but cannot cause infinite loading. default "load"
pageWaitSelectorCategory	string	For what selector to wait on each page before extracting data
pageWaitSelectorArticle	string	For what selector to wait on each page before extracting data
scrollToBottom	boolean	Scroll to the bottom of the page, loading dynamic articles.
scrollToBottomButtonSelector	string	CSS selector for a button to load more articles
scrollToBottomMaxSecs	integer	Limit for how long the scrolling can run so it does not go infinite.
extendOutputFunction	string	This function allows you to merge your custom extraction with the default one. You can only return an object from this function. This object will be merged/overwritten with the default output for each article.
stopAfterCUs	integer	The scraper will stop running after reaching this number of compute units.
notificationEmails	array	Notifications will be sent to these email addresses.
notifyAfterCUs	integer	The scraper will send notifications to the provided email when it reaches this number of CUs.

Call it

curl

curl https://cracked-ai-pearl.vercel.app/v1/run \
  -H "Authorization: Bearer ck_live_..." -H "content-type: application/json" \
  -d '{"provider":"apify","endpoint":"/lukaskrivka/article-extractor-smart","input":{"startUrls":[{"url":"https://www.theguardian.com"}],"articleUrls":[{"url":"https://www.bbc.com/news/uk-62836057"}],"onlyNewArticles":false,"onlyNewArticlesPerDomain":false,"onlyInsideArticles":true,"enqueueFromArticles":false,"crawlWholeSubdomain":false,"onlySubdomainArticles":false,"scanSitemaps":false,"sitemapUrls":[{"url":"https://www.theguardian.com/sitemaps/news.xml"}],"saveHtml":false,"saveHtmlAsLink":false,"saveSnapshots":false,"useGoogleBotHeaders":false,"minWords":150,"dateFrom":"2024-01-01","onlyArticlesForLastDays":7,"mustHaveDate":true,"isUrlArticleDefinition":{"minDashes":4,"hasDate":true,"linkIncludes":["article","storyid","?p=","id=","/fpss/track",".html","/content/"]},"pseudoUrls":[{"purl":"https://www.theguardian.com/technology/[.*]"}],"linkSelector":"a.article-link","maxDepth":3,"maxPagesPerCrawl":100,"maxArticlesPerCrawl":3,"maxArticlesPerStartUrl":3,"maxConcurrency":3,"proxyConfiguration":{"useApifyProxy":true},"overrideProxyGroup":"SHADER","useBrowser":false,"pageWaitMs":1000,"navigationWaitUntil":"load","pageWaitSelectorCategory":".article-list","pageWaitSelectorArticle":".article-body","scrollToBottom":false,"scrollToBottomButtonSelector":".load-more-button","scrollToBottomMaxSecs":30,"extendOutputFunction":"($) => {\n    const result = {};\n    // Uncomment to add a title to the output\n    // result.pageTitle = $('title').text().trim();\n\n    return result;\n}","stopAfterCUs":10,"notificationEmails":["user@example.com"],"notifyAfterCUs":5,"notifyAfterCUsPeriodically":5},"wait":false}'

poll

curl "https://cracked-ai-pearl.vercel.app/v1/runs/<runId>?wait=30" -H "Authorization: Bearer ck_live_..."

cli

npx cracked-ai run -p apify -e /lukaskrivka/article-extractor-smart -i '{"startUrls":[{"url":"https://www.theguardian.com"}],"articleUrls":[{"url":"https://www.bbc.com/news/uk-62836057"}],"onlyNewArticles":false,"onlyNewArticlesPerDomain":false,"onlyInsideArticles":true,"enqueueFromArticles":false,"crawlWholeSubdomain":false,"onlySubdomainArticles":false,"scanSitemaps":false,"sitemapUrls":[{"url":"https://www.theguardian.com/sitemaps/news.xml"}],"saveHtml":false,"saveHtmlAsLink":false,"saveSnapshots":false,"useGoogleBotHeaders":false,"minWords":150,"dateFrom":"2024-01-01","onlyArticlesForLastDays":7,"mustHaveDate":true,"isUrlArticleDefinition":{"minDashes":4,"hasDate":true,"linkIncludes":["article","storyid","?p=","id=","/fpss/track",".html","/content/"]},"pseudoUrls":[{"purl":"https://www.theguardian.com/technology/[.*]"}],"linkSelector":"a.article-link","maxDepth":3,"maxPagesPerCrawl":100,"maxArticlesPerCrawl":3,"maxArticlesPerStartUrl":3,"maxConcurrency":3,"proxyConfiguration":{"useApifyProxy":true},"overrideProxyGroup":"SHADER","useBrowser":false,"pageWaitMs":1000,"navigationWaitUntil":"load","pageWaitSelectorCategory":".article-list","pageWaitSelectorArticle":".article-body","scrollToBottom":false,"scrollToBottomButtonSelector":".load-more-button","scrollToBottomMaxSecs":30,"extendOutputFunction":"($) => {\n    const result = {};\n    // Uncomment to add a title to the output\n    // result.pageTitle = $('title').text().trim();\n\n    return result;\n}","stopAfterCUs":10,"notificationEmails":["user@example.com"],"notifyAfterCUs":5,"notifyAfterCUsPeriodically":5}' -w 120

mcp

run_tool({ provider: "apify", endpoint: "/lukaskrivka/article-extractor-smart", input: {"startUrls":[{"url":"https://www.theguardian.com"}],"articleUrls":[{"url":"https://www.bbc.com/news/uk-62836057"}],"onlyNewArticles":false,"onlyNewArticlesPerDomain":false,"onlyInsideArticles":true,"enqueueFromArticles":false,"crawlWholeSubdomain":false,"onlySubdomainArticles":false,"scanSitemaps":false,"sitemapUrls":[{"url":"https://www.theguardian.com/sitemaps/news.xml"}],"saveHtml":false,"saveHtmlAsLink":false,"saveSnapshots":false,"useGoogleBotHeaders":false,"minWords":150,"dateFrom":"2024-01-01","onlyArticlesForLastDays":7,"mustHaveDate":true,"isUrlArticleDefinition":{"minDashes":4,"hasDate":true,"linkIncludes":["article","storyid","?p=","id=","/fpss/track",".html","/content/"]},"pseudoUrls":[{"purl":"https://www.theguardian.com/technology/[.*]"}],"linkSelector":"a.article-link","maxDepth":3,"maxPagesPerCrawl":100,"maxArticlesPerCrawl":3,"maxArticlesPerStartUrl":3,"maxConcurrency":3,"proxyConfiguration":{"useApifyProxy":true},"overrideProxyGroup":"SHADER","useBrowser":false,"pageWaitMs":1000,"navigationWaitUntil":"load","pageWaitSelectorCategory":".article-list","pageWaitSelectorArticle":".article-body","scrollToBottom":false,"scrollToBottomButtonSelector":".load-more-button","scrollToBottomMaxSecs":30,"extendOutputFunction":"($) => {\n    const result = {};\n    // Uncomment to add a title to the output\n    // result.pageTitle = $('title').text().trim();\n\n    return result;\n}","stopAfterCUs":10,"notificationEmails":["user@example.com"],"notifyAfterCUs":5,"notifyAfterCUsPeriodically":5}, wait: false })

Try it

Runs against your signed-in workspace balance. Sign in if you have not.

{
  "startUrls": [
    {
      "url": "https://www.theguardian.com"
    }
  ],
  "articleUrls": [
    {
      "url": "https://www.bbc.com/news/uk-62836057"
    }
  ],
  "onlyNewArticles": false,
  "onlyNewArticlesPerDomain": false,
  "onlyInsideArticles": true,
  "enqueueFromArticles": false,
  "crawlWholeSubdomain": false,
  "onlySubdomainArticles": false,
  "scanSitemaps": false,
  "sitemapUrls": [
    {
      "url": "https://www.theguardian.com/sitemaps/news.xml"
    }
  ],
  "saveHtml": false,
  "saveHtmlAsLink": false,
  "saveSnapshots": false,
  "useGoogleBotHeaders": false,
  "minWords": 150,
  "dateFrom": "2024-01-01",
  "onlyArticlesForLastDays": 7,
  "mustHaveDate": true,
  "isUrlArticleDefinition": {
    "minDashes": 4,
    "hasDate": true,
    "linkIncludes": [
      "article",
      "storyid",
      "?p=",
      "id=",
      "/fpss/track",
      ".html",
      "/content/"
    ]
  },
  "pseudoUrls": [
    {
      "purl": "https://www.theguardian.com/technology/[.*]"
    }
  ],
  "linkSelector": "a.article-link",
  "maxDepth": 3,
  "maxPagesPerCrawl": 100,
  "maxArticlesPerCrawl": 3,
  "maxArticlesPerStartUrl": 3,
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "overrideProxyGroup": "SHADER",
  "useBrowser": false,
  "pageWaitMs": 1000,
  "navigationWaitUntil": "load",
  "pageWaitSelectorCategory": ".article-list",
  "pageWaitSelectorArticle": ".article-body",
  "scrollToBottom": false,
  "scrollToBottomButtonSelector": ".load-more-button",
  "scrollToBottomMaxSecs": 30,
  "extendOutputFunction": "($) => {\n    const result = {};\n    // Uncomment to add a title to the output\n    // result.pageTitle = $('title').text().trim();\n\n    return result;\n}",
  "stopAfterCUs": 10,
  "notificationEmails": [
    "user@example.com"
  ],
  "notifyAfterCUs": 5,
  "notifyAfterCUsPeriodically": 5
}