#1483: Enabling indexing of the public knowledge base portal for LLM (llms.txt)

Отредактирована: 92 дня назад

Symptoms

How to enable indexing of the public knowledge base portal for LLMs (large language models) such as Deepseek, Gemini, ChatGPT, Perplexity, etc.?

Does Swarmica support the llms.txt standard?

Solution

To enable the generation of llms.txt files, you need to update Swarmica to version v5.9.0 or higher and connect the knowledge base portal service.
Add the feature flag __llmstxt to FEATURES in the /root/swarmica/.env file.
Check that the rule directive for the helpcenter service in /root/swarmica/docker-compose.yml contains ||Path(`/llms.txt`):

helpcenter:
    image: reg.gl.swd.im/swarmica/helpcenter:${SW_HELPCENTER_VERSION}
    restart: always
    depends_on:
      - django
    env_file:
      - .env
    volumes:
      - .env:/app/.env
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.helpcenter.loadbalancer.server.port=4000"
      - "traefik.http.routers.helpcenter-router.rule=Host(`$SW_HOSTNAME`) && (PathPrefix(`/category`)||PathRegexp(`^/article/.*$`)||PathPrefix(`/_next`)||PathPrefix(`/help`)||Path(`/robots/sitemap.xml`)||Path(`/llms.txt`))"
      - "traefik.http.routers.helpcenter-router.priority=11"
      - "traefik.http.routers.helpcenter-router.entrypoints=web-secure"
      - "traefik.http.routers.helpcenter-router.middlewares=secure-headers,compression"
      - "traefik.http.routers.helpcenter-router.service=helpcenter"
      - "traefik.http.routers.helpcenter-router.tls.certresolver=letsencrypt"

Restart the application:

sudo su -
cd /root/swarmica
docker compose down && docker compose up -d

Additional information

At the moment, there is no single standard used by all large language models. However, many LLMs parse llms.txt and try to extract data about the information available on the site from it.

Swarmica generates an llms.txt file using the names of the Knowledge Base sections, article titles, and links to their Markdown version, which is preferred for LLM.