katana

katana is a command-line web crawler and spidering tool written in Go by ProjectDiscovery. It takes target URLs from a flag, a list file, or STDIN, and crawls them in either standard mode (plain HTTP requests) or headless hybrid mode (a Chrome instance, with options such as -system-chrome, -show-browser, -no-sandbox, and connecting to an existing browser via -chrome-ws-url). Output goes to STDOUT, a file, or JSON, which makes it usable as a stage in automation pipelines.

Crawling behavior is controlled through a large set of flags. Depth defaults to 3, and the visit strategy can be depth-first or breadth-first. The tool can parse JavaScript files for endpoints (-js-crawl, plus an optional memory-intensive -jsluice mode), crawl known files such as robots.txt and sitemap.xml, fill forms automatically (experimental), extract form elements and XHR requests into JSONL output, and detect technologies. It also offers URL deduplication options: ignoring differing query parameters, filtering similar-looking paths such as /users/123 and /users/456, and page-content similarity filtering using simhash, tfidf, or bm25. A knowledge base feature classifies pages and forms with an auto-downloaded ML model, and can optionally extract secrets and classify REST, GraphQL, SOAP, and XHR endpoints.

Scope is set with the field scope (dn, rdn, fqdn, or a custom regex) or with in-scope and out-of-scope URL regexes. Output can be narrowed with regex match/filter, extension match/filter, and DSL-based match and filter conditions on responses. Rate limiting covers concurrency, parallelism, per-second and per-minute request caps, and per-host limits. Proxy support (HTTP/SOCKS5), custom headers and cookies, custom resolvers, TLS client hello (JA3) randomization, and captcha solver integration are also documented.

It is aimed at security researchers, bug bounty hunters, and penetration testers who need to enumerate endpoints and assets of web applications, as well as developers and DevOps engineers who want to script site crawling. It installs with go install (Go 1.26+ is required) or from prebuilt release binaries, and a Docker image is available.

Installation

# via Go
CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest

# via Docker
docker pull projectdiscovery/katana:latest