Crawler Tests

HTTP Header X-Robots-Tag all directives for all bots

X-Robots-Tag = max-snippet:20,noarchive,nositelinkssearchbox,nosnippet,indexifembedded,max-image-preview:9,max-video-preview:12,notranslate,noimageindex,unavailable_after:2023-12-23