txt
CopyEdit
User-agent: *
Disallow: /cache/
Disallow: /index.php/index/oai
Allow: /
Sitemap: https://matjournals.net/sitemap.xml
Disallow: /cache/ — prevents bots from crawling cached files.
Disallow: /index.php/index/oai — optional if you're not using OAI-PMH crawling.
Allow: / — ensures your content is generally crawlable.
Sitemap: — helps bots discover your content if you have a sitemap.