How to Index Your Next.js Apps Faster Using IndexNow

Next.js is a powerful framework for building lightning-fast applications. However, getting these applications indexed quickly by search engines is crucial for visibility and traffic, and sadly, this is not immediate.

Even after uploading your sitemap, it can take up to weeks or even months before search engines crawl your pages. In fact, if you have updated or added any new page, it can take weeks for the search engines to notice.

So, can we do any better?

In this article, you'll learn how to boost your Next.js app's SEO on major search engines like Bing, Yahoo, and so on with fast indexing using IndexNow.

Table of Contents

• What is IndexNow?
  • How Does it Work?
• Steps
  • Prerequisites
  • High Level Steps
  • How to Prove Ownership of the Host
  • How to Create a Script to Get All URLs from Your Sitemap
  • How to Call the IndexNow API
• Next Steps and Improvements
• Conclusion
• Links and References

What is IndexNow?

IndexNow is a protocol that drastically reduces indexing time. Here's how it's defined on their website:

IndexNow is an easy way for websites owners to instantly inform search engines about latest content changes on their website. In its simplest form, IndexNow is a simple ping so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.(Source: IndexNow Home)

It's a protocol that is adopted by the likes of Bing, Naver, Seznam.cz, Yandex, Yep and others. Google does not support this protocol as of the writing of this article.

This is natively integrated into many CMS like Wix, and there are many third party plugins for others like Drupal or WordPress. However, there is no native support in NextJS.

How Does it Work?

Every time you update something, all you need to do is "ping" or call their API and inform them of the change.

When this information is received, the search engines can now prioritize crawling these URLs over the other URLs that are being crawled naturally.

In this guide, we'll walk through the process of integrating IndexNow into your existing Next.js app so that any changes to URLs can be submitted and indexed by the search engines.

Prerequisites

• A Next.js app.
• A sitemap.xml for your Next.js app.

High Level Steps

1. We first need to "prove ownership" of the host for which URLs will be submitted.
2. Create a simple Node.js script to get all URLs from your sitemap.
3. Call the IndexNow API.

How to Prove Ownership of the Host

Go to the IndexNow page for Bing. Since there is no direct integration for Next.js, scroll down to the section of manual integration.

Click on "Generate" to generate a new API Key.

In your Next.js app, go to the public directory in your root. All the static content is rendered through this directory. Create a new file and store this API key:

# Assuming API Key is "f34f184d10c049ef99aa7637cdc4ef04". Change according to yourr generated API Key
echo "f34f184d10c049ef99aa7637cdc4ef04" > f34f184d10c049ef99aa7637cdc4ef04.txt

Build and run your Next.js app:

npm run build && npm run start

Then confirm that the file is available in your path /f34f184d10c049ef99aa7637cdc4ef04.txt.

That is, opening https://localhost:3000/f34f184d10c049ef99aa7637cdc4ef04.txt should give a response with text "f34f184d10c049ef99aa7637cdc4ef04" on your browser.

Depending on your API Key, modify the above key value. Commit, push and deploy this changes to your production.

On successful deployment, verify that <Your URL>/<API Key>.txt renders <API Key> text. That is: <Your URL>/f34f184d10c049ef99aa7637cdc4ef04.txt should render f34f184d10c049ef99aa7637cdc4ef04.

How to Create a Script to Get All URLs from Your Sitemap

First, create the Node script file:

touch lib/indexnow.js

Then add the code below:

const xml2js = require('xml2js');
// Configuration
const sitemapUrl = '<Your URL>/sitemap.xml'; // TODO: Update
const host = '<Your URL>'; // TODO: Update
const key = '<API Key>'; // TODO: Update
const keyLocation = 'https://<Your URL>/<API Key>.txt'; // TODO: Update
const modifiedSinceDate = new Date(process.argv[2] || '1970-01-01');
if (isNaN(modifiedSinceDate.getTime())) {
  console.error('Invalid date provided. Please use format YYYY-MM-DD');
  process.exit(1);
}
function fetchSitemap(url) {
  return new Promise((resolve, reject) => {
    https.get(url, (res) => {
      let data = '';
      res.on('data', (chunk) => {
        data += chunk;
      });
      res.on('end', () => {
        resolve(data);
      });
    }).on('error', (err) => {
      reject(err);
    });
  });
}
function parseSitemap(xmlData) {
  return new Promise((resolve, reject) => {
    xml2js.parseString(xmlData, (err, result) => {
      if (err) {
        reject(err);
      } else {
        resolve(result);
      }
    });
  });
}
function filterUrlsByDate(sitemap, date) {
  const urls = sitemap.urlset.url;
  return urls
    .filter(url => new Date(url.lastmod[0]) > date)
    .map(url => url.loc[0]);
}
async function main() {
  try {
    const xmlData = await fetchSitemap(sitemapUrl);
    const sitemap = await parseSitemap(xmlData);
    const filteredUrls = filterUrlsByDate(sitemap, modifiedSinceDate);
    console.log(filteredUrls);
  } catch (error) {
    console.error('An error occurred:', error);
  }
}
main();

We basically fetch URLs from the sitemap that have been modified from a certain date. The date can be passed as an argument to the script.

Next, install the xml2js library which we'll use to parse the XML response from sitemap.

npm install xml2js --save-dev

Then run the script to fetch URLs and check if everything works:

node lib/indexnow.js 2024-01-01

This should output a list of URLs that have been modified since 1 Jan, 2024. You can pass any date to it.

How to Call the IndexNow API

Here's the IndexNow API Schema:

Request:

POST /IndexNow HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: api.indexnow.org
{
  "host": "www.example.org",
  "key": "7e3f6e8bc47b4f2380ba54aab6088521",
  "keyLocation": "https://www.example.org/7e3f6e8bc47b4f2380ba54aab6088521.txt",
  "urlList": [
      "https://www.example.org/url1",
      "https://www.example.org/folder/url2",
      "https://www.example.org/url3"
      ]
}

Response:

Now that we are certain that our URL fetching portion works correctly, let's add the main function to call the IndexNow API:

Open the lib/index.js file using your favorite IDE and add the following function:

function postToIndexNow(urlList) {
  const data = JSON.stringify({
    host,
    key,
    keyLocation,
    urlList
  });
  const options = {
    hostname: 'api.indexnow.org',
    port: 443,
    path: '/IndexNow',
    method: 'POST',
    headers: {
      'Content-Type': 'application/json; charset=utf-8',
      'Content-Length': data.length
    }
  };
  return new Promise((resolve, reject) => {
    const req = https.request(options, (res) => {
      let responseData = '';
      res.on('data', (chunk) => {
        responseData += chunk;
      });
      res.on('end', () => {
        resolve({
          statusCode: res.statusCode,
          statusMessage: res.statusMessage,
          data: responseData
        });
      });
    });
    req.on('error', (error) => {
      reject(error);
    });
    req.write(data);
    req.end();
  });
}

This function makes the call to the IndexNow API by passing a list of URLs that is passed to it, along with the <API Key>.

Call this function from the main function. Modify the main function as following:

async function main() {
  try {
    const xmlData = await fetchSitemap(sitemapUrl);
    const sitemap = await parseSitemap(xmlData);
    const filteredUrls = filterUrlsByDate(sitemap, modifiedSinceDate);
    console.log(filteredUrls);
    if (filteredUrls.length > 0) {
      const response = await postToIndexNow(filteredUrls);
      console.log('IndexNow API Response:');
      console.log('Status:', response.statusCode, response.statusMessage);
      console.log('Data:', response.data);
    } else {
      console.log('No URLs modified since the specified date.');
    }
  } catch (error) {
    console.error('An error occurred:', error);
  }
}

The IndexNow API will now be called for every URL that we pass to it.

Run the script, and you should see a similar output on success, or an error message in case of any issues:

% node lib/indexnow.js 2024-07-24
[
  'https://<Your URL>',
  'https://<Your URL>/page1',
  'https://<Your URL>/page1'
]
IndexNow API Response:
Status: 200 OK
Data:

Voila, your APIs can now work with IndexNow for faster indexing.

Next Steps and Improvements

We just looked at how to write and execute a script locally to index our pages via IndexNow. However, there are many things that can be done to improve this further, for instance:

• Integrate IndexNow into your CI/CD pipeline for automatic updates.
• Handle large sitemaps efficiently with batching or queuing.
• Monitor and log IndexNow submissions for debugging and analytics.
• Explore the IndexNow API for additional functionalities (for example: URL deletion).
• Provide CLI version.
• Add TypeScript support.

However, these are out of scope for this article. There are few production ready npm modules that implement some or more of these advanced behaviors that can be easily integrated into your app. I have created one (see indexnow-submitter announcement) as well as adding missing features/support in the ecosystem. You can plug and use any of these modules in your Node based applications.

Conclusion

In this article, you learned how to add the IndexNow protocol to your Next.js application. You can leverage this protocol to get a much faster, automated and convenient page indexing experience whenever you make any change to your website, allowing search engines to fetch the latest content from your pages.

I hope this was useful, and feel free to experiment and customize this integration further to suit your needs.

Links and References

• IndexNow Documentation
• FAQ
• IndexNow Bing
• indexnow-submitter NPM module
• indexnow-submitter release announcement