http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/ Updates on the web crawler behavior research project. en-us 2026-02-19T12:00:00Z http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html The crawler behavior research site is now live with files of various types. 2026-02-19T12:00:00Z research-2026-02-19 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html PNG, JPEG, GIF, SVG, and WebP images have been added to test crawler behavior with binary content. 2026-02-18T12:00:00Z research-2026-02-18 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html S3 server access logging is now active, capturing timestamp, referrer, and user-agent for every request. 2026-02-17T12:00:00Z research-2026-02-17 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html An XML sitemap has been published listing all files. Observing which crawlers discover and parse it. 2026-02-16T12:00:00Z research-2026-02-16 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html Substantial JavaScript and CSS files added to test whether crawlers download and parse non-HTML resources. 2026-02-15T12:00:00Z research-2026-02-15 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html Custom robots.txt with per-crawler directives published. Monitoring compliance across different bots. 2026-02-14T12:00:00Z research-2026-02-14 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html JSON and XML data files added. Testing whether crawlers index structured data formats. 2026-02-13T12:00:00Z research-2026-02-13 http://site4726837462198733423.s3-website-us-east-1.amazonaws.com/about.html A configuration file without extension has been added to test Content-Type handling by crawlers. 2026-02-12T12:00:00Z research-2026-02-12