#2149: sitemap.xml and robots.txt for the public knowledge base
Отредактирована: сегодняA sitemap (sitemap.xml) tells search engines which pages of the public knowledge base should be indexed. The robots.txt file sets the rules for crawling the site by search robots.
Where sitemap.xml is located
The sitemap is generated by the help center service (helpcenter). It is available at:
https://<your Swarmica domain>/robots/sitemap.xml
The help center must be enabled and configured — see Configuring the SEO-optimized public knowledge base portal.
What is included in sitemap.xml
- The knowledge base home page
/help. - Published articles with All access. The articles are sorted by the number of linked tickets: the most popular come first.
- Knowledge base categories that contain articles from the sitemap.
Each language of an article is listed as a separate address with hreflang links to the other language versions. The lastmod date corresponds to the date of the last article change.
The sitemap is cached, so new and changed articles appear in it with a short delay.
Number of articles in sitemap.xml
By default, the sitemap includes 500 articles. The number is set by a variable in the /root/swarmica/.env file:
# The number of published articles with All access in /robots/sitemap.xml
NEXT_PUBLIC_SITEMAPXML_MAX_ARTICLES=500
The value of NEXT_PUBLIC_SITEMAPXML_MAX_ARTICLES must not exceed the value of the MAX_PAGINATION_LIMIT variable. If you set more than 500 articles, increase MAX_PAGINATION_LIMIT as well.
After changing .env, restart the application:
cd /root/swarmica
docker compose down
docker compose up -d
Configuring robots.txt
The robots.txt file is located on the Swarmica server at /root/swarmica/robots.txt. By default, it allows indexing of articles, closes service sections and specifies the sitemap address:
User-Agent: *
Disallow: /attachments/
Disallow: /api/
Disallow: /category/
Disallow: /help/
# Add below line if you want your API schema to be indexed by robots
#Allow: /api/schema/doc/
sitemap: https://YOUR_SWARMICA_DOMAIN/robots/sitemap.xml
Replace YOUR_SWARMICA_DOMAIN with the domain of your knowledge base.
If Swarmica is available under several domain names, specify the sitemap address for each of them in robots.txt:
sitemap: https://YOUR_SWARMICA_DOMAIN1/robots/sitemap.xml
sitemap: https://YOUR_SWARMICA_DOMAIN2/robots/sitemap.xml
sitemap: https://YOUR_SWARMICA_DOMAIN3/robots/sitemap.xml
To disable indexing of the knowledge base completely, keep only two lines in the file:
User-Agent: *
Disallow: /
Submitting the sitemap to search engines
To speed up indexing, add the sitemap address in the webmaster tools:
- Yandex Webmaster — Indexing → Sitemap files;
- Google Search Console — Sitemaps.