Krptkn.Prelaunch (Кропоткин Framework v0.1.0) View Source

Useful functions to run on the starting URL, so the URL queue fills up faster. This will also help discover non-linked pages of a website.

Link to this section Summary

Functions

dictionary(uri)

Provides a list of common URLs to check.

robotstxt(uri, text)

Checks the pages listed on Robots.txt.

Link to this section Functions

dictionary(uri)

Provides a list of common URLs to check.

Returns a list of strings.

Examples

iex> Krptkn.Prelaunch.dictionary("https://stallman.org/")
[
  "https://stallman.org/wp-content/",
  "https://stallman.org/wp-content/plugins/",
  "https://stallman.org/uploads/",
]

robotstxt(uri, text)

Checks the pages listed on Robots.txt.

Returns a list of strings.

Examples

iex> Krptkn.Prelaunch.robotstxt("https://stallman.org/", "Example robots.txt file data")
[
  "https://stallman.org/admin/",
  "https://stallman.org/private-images/",
]