1. Forum
  2. FidoNet
  3. MYSTIC

  • File base with FTP Name with dot in it?

    From mark lewis@1:3634/12.73 to Björn Wiberg on Tue Apr 5 08:44:38 2022

    On 2022 Mar 20 21:21:42, you wrote to g00r00:

    I just noticed that Googlebot has begun crawling the anonymous FTP here. :-D

    To my surprise, they seem to try to enter a /robots.txt subdirectory before
    trying to download /robots.txt:

    that's how they decide if it is a file or a directory... if it is a file, then they download it...

    )\/(ark

    "The soul of a small kitten in the body of a mighty dragon. Look on my majesty, ye mighty, and despair! Or bring me catnip. Your choice. Oooh, a shiny thing!"
    ... Cats are magical, the more you pet them the longer you both live
    ---
    * Origin: (1:3634/12.73)
  • From mark lewis@1:3634/12.73 to Björn Wiberg on Wed Apr 6 13:45:10 2022

    On 2022 Apr 05 16:24:54, you wrote to me:

    To my surprise, they seem to try to enter a /robots.txt subdirectory

    that's how they decide if it is a file or a directory... if it is a
    file, then they download it...

    Yes, I suspected that might be the case... I wonder, though, if they
    do anything special if the cd succeeds

    if the cd succeeds they do an ls to see if there's file and directories in it... if there are, they download them and continue walking through the directories...

    or if they just conclude "not a file" and skip it. (According to the
    spec, they shouldn't look for anything inside that directory, if it exists.)

    what spec is that?

    )\/(ark

    "The soul of a small kitten in the body of a mighty dragon. Look on my majesty, ye mighty, and despair! Or bring me catnip. Your choice. Oooh, a shiny thing!"
    ... If all you have is a hammer, everything looks like a nail.
    ---
    * Origin: (1:3634/12.73)
  • From mark lewis@1:3634/12.73 to Björn Wiberg on Fri Apr 8 06:46:44 2022

    On 2022 Apr 07 09:50:02, you wrote to me:

    http://example.com/folder/robots.txt
    "Not a valid robots.txt file. Crawlers don't check for robots.txt files in subdirectories. "

    this is true... robots.txt applies to the entire site so it is only valid when found in the root directory of the site... if it is located anywhere else, it is just another file...

    )\/(ark

    "The soul of a small kitten in the body of a mighty dragon. Look on my majesty, ye mighty, and despair! Or bring me catnip. Your choice. Oooh, a shiny thing!"
    ... People build walls to see who cares enough to break them down.
    ---
    * Origin: (1:3634/12.73)