At first I kept focusing on the main site without noticing that the name of the problem hints at the robots.txt file itself.

Simply access robots.txt and it will show you this:

User-agent: *
Disallow: /cc6b1.html

Go to the disallowed path and you’ll get the flag