In this article, I will tell you how to automatically transmit information about new, changed or deleted content to search engines, and make site indexing almost instantaneous.
Setting up IndexNow
Yandex and Bing use an open license protocol to quickly notify search engines about changes on the site. Before sending requests, it is necessary to confirm the rights to the site through a special key. In the future, data will be exchanged via the API.
Yes, it is advisable to register your site in Bing Webmaster Tools (BWT) and use fast indexing. In some niches in Runet from Bing there is traffic and conversions.
The setup process is identical for both search engines:
Generate a key. This is a random set of characters exclusively from the ranges a-z, A-Z, 0-9. The key length ranges from 8 to 128 characters.
Create a file with the key and put it in the root of the site. Creating a txt file in UTF-8 encoding. We use the key as the name and content. Example: 76934bb5f78b405dac82129a90a6ec51.txt . We fill the file into the root directory where all the files of the site are located. The file thereby confirms the rights to the site. You can place a file in a specific site category. Then sending requests for quick indexing will be carried out for URLs from this category only.
We send the URL for indexing to Yandex. Request example:
- We send the URL for indexing to Bing.
- You can also configure scanning via IndexNow via plugins for popular CMS:
- 1C-Bitrix – Picom, Reindex;
- WordPress – IndexNow;
- UMI.CMS – IndexNow;
- Moguta.CMS – IndexNow.
Up to 10,000 URLs can be given priority indexing in one request. Spam is not worth it, search engines can limit the quota.
It is also not recommended to send old pages, especially if they are already in the index. This entails an irrational use of the power of search engines.
It is worth noting that using the IndexNow protocol does not guarantee 100% hit in the index of a particular page. This only indicates to search robots the prioritization of URL indexing, and does not cancel the requirements for the quality of the page and content.
A few years ago, Yandex had problems with this, it was reluctant to index new URLs and update old ones via the IndexNow protocol. Now, in practice, high-quality pages quite quickly get into the Yandex and Bing index and begin to rank within a day.
Configuring the Google Indexing API
Indexing API allows you to send new, updated or deleted URLs for priority indexing in Google. This method works much faster than automatic updates via sitemap.xml .
The Google robot does not scan often sitemap.xml for the presence of changes, so new pages may not be ranked for months.
The setup process consists of the following blocks: creating a service account, obtaining a JSON key, connecting the Indexing API to GSC, installing node.js and sending URLs for indexing.
Creating a service account. Go to Google Cloud Platform and click Create Project. Enter an arbitrary project name. Then click Create Service Account with the Owner role.
We get the JSON key. Click on the three dots in the Actions column and select Manage keys, then Add key -> Create new key. Download the file with the JSON key.
Linking Indexig API and GSC. In GSC, go to Settings -> Users and Permissions, click Add User. We take the e-mail of the service account and provide full access. Go back to GCP and enable the Indexing API via the Enable button.
Install node.js . Download it for your OS here and install it. Then we download the script for mass URL sending via node.js from GitHub. Next, you need to open a command prompt or terminal, go to the folder with the script (cd command * full path to the folder*) and execute 3 commands: npm install requests, npm audit fix, npm audit fix –force. After successful execution, the node_modules folder will appear in the script folder.
We send the URL via the Google Indexing API. Go to the folder with the script from the paragraph above, Open the service_account file.json and copy the contents of the JSON key from the paragraph above into it. To a file urls.txt adding our URLs that we want to index. Next, you need to open a command prompt or terminal, go to the folder with the script (cd command * full path to the folder*) and run the command: node index.js.
You can also send URLs for indexing via CMS plugins. There are quite a few of them. To do this, you will still have to create a service account and get a JSON key.
The quota in Google is 200 URLs per day and 100 URLs per upload. Also, as in IndexNow, this method does not guarantee 100% indexing at the same second. We just set the highest indexing priority for Googlebot, and then it acts at its discretion.
The screenshot below shows an example of using the Indexing API for a small site.
Indexing status in GSC
As content was added, all new URLs were immediately sent for indexing via the Indexing API.
Manual priority indexing of pages
Yandex.Webmaster – go to Indexing -> Page Reindexing.
GSC – enter the URL in the top line of the URL check and click Request Indexing.
BWT – go to the section Sending URLs, click Send URLs.
Bonuses from fast site indexing
Important money pages get into the index faster, content is updated faster, unnecessary URLs are deleted faster.
Problems with identifying the original source site and stealing content are minimized.
Mass initial addition of pages to the index allows you to quickly increase traffic from search engines.