Skip to content

Commit

Permalink
chore: -
Browse files Browse the repository at this point in the history
  • Loading branch information
enenumxela committed Oct 12, 2023
1 parent 15c21eb commit c4266b7
Show file tree
Hide file tree
Showing 26 changed files with 392 additions and 266 deletions.
6 changes: 1 addition & 5 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
# Executable

bin

# Notes

notes.txt
bin
183 changes: 141 additions & 42 deletions .golangci.yaml
Original file line number Diff line number Diff line change
@@ -1,61 +1,160 @@
run:
issues-exit-code: 1
# Timeout for analysis, e.g. 30s, 5m.
# Default: 1m
timeout: 5m

linters:
# Disable all linters.
# Default: false
disable-all: true
# Enable specific linter
enable:
- bodyclose
- depguard
- dogsled
- dupl
- errcheck
- exportloopref
- exhaustive
- goconst
- gocritic
- gofmt
- goimports
- gocyclo
- gosec
- gosimple
- govet
- ineffassign
- misspell
- nolintlint
- prealloc
- predeclared
- revive
- staticcheck
- stylecheck
- thelper
- tparallel
- typecheck
- unconvert
- unparam
- unused
- whitespace
- wsl
# Enabled by Default
- errcheck # errcheck is a program for checking for unchecked errors in Go code. These unchecked errors can be critical bugs in some cases [fast: false, auto-fix: false]
- gosimple # (megacheck) # Linter for Go source code that specializes in simplifying code [fast: false, auto-fix: false]
- govet # (vet, vetshadow) # Vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string [fast: false, auto-fix: false]
- ineffassign # Detects when assignments to existing variables are not used [fast: true, auto-fix: false]
- staticcheck # (megacheck) # It's a set of rules from staticcheck. It's not the same thing as the staticcheck binary. The author of staticcheck doesn't support or approve the use of staticcheck as a library inside golangci-lint. [fast: false, auto-fix: false]
- unused # (megacheck) # Checks Go code for unused constants, variables, functions and types [fast: false, auto-fix: false]
# Disabled by Default
- asasalint # check for pass []any as any in variadic func(...any) [fast: false, auto-fix: false]
- asciicheck # Simple linter to check that your code does not contain non-ASCII identifiers [fast: true, auto-fix: false]
- bidichk # Checks for dangerous unicode character sequences [fast: true, auto-fix: false]
- bodyclose # checks whether HTTP response body is closed successfully [fast: false, auto-fix: false]
- containedctx # containedctx is a linter that detects struct contained context.Context field [fast: false, auto-fix: false]
- contextcheck # check whether the function uses a non-inherited context [fast: false, auto-fix: false]
# - cyclop # checks function and package cyclomatic complexity [fast: false, auto-fix: false]
# - deadcode # [deprecated] # Finds unused code [fast: false, auto-fix: false]
- decorder # check declaration order and count of types, constants, variables and functions [fast: true, auto-fix: false]
# - depguard # Go linter that checks if package imports are in a list of acceptable packages [fast: true, auto-fix: false]
- dogsled # Checks assignments with too many blank identifiers (e.g. x, _, _, _, := f()) [fast: true, auto-fix: false]
# - dupl # Tool for code clone detection [fast: true, auto-fix: false]
- dupword # checks for duplicate words in the source code [fast: true, auto-fix: true]
- durationcheck # check for two durations multiplied together [fast: false, auto-fix: false]
- errchkjson # Checks types passed to the json encoding functions. Reports unsupported types and optionally reports occasions, where the check for the returned error can be omitted. [fast: false, auto-fix: false]
- errname # Checks that sentinel errors are prefixed with the `Err` and error types are suffixed with the `Error`. [fast: false, auto-fix: false]
- errorlint # errorlint is a linter for that can be used to find code that will cause problems with the error wrapping scheme introduced in Go 1.13. [fast: false, auto-fix: false]
- execinquery # execinquery is a linter about query string checker in Query function which reads your Go src files and warning it finds [fast: false, auto-fix: false]
- exhaustive # check exhaustiveness of enum switch statements [fast: false, auto-fix: false]
# - exhaustivestruct # [deprecated] # Checks if all struct's fields are initialized [fast: false, auto-fix: false]
# - exhaustruct # Checks if all structure fields are initialized [fast: false, auto-fix: false]
- exportloopref # checks for pointers to enclosing loop variables [fast: false, auto-fix: false]
# - forbidigo # Forbids identifiers [fast: false, auto-fix: false]
- forcetypeassert # finds forced type assertions [fast: true, auto-fix: false]
# - funlen # Tool for detection of long functions [fast: true, auto-fix: false]
- gci # Gci controls Go package import order and makes it always deterministic. [fast: true, auto-fix: false]
- ginkgolinter # enforces standards of using ginkgo and gomega [fast: false, auto-fix: false]
- gocheckcompilerdirectives # Checks that go compiler directive comments (//go:) are valid. [fast: true, auto-fix: false]
# - gochecknoglobals # check that no global variables exist [fast: false, auto-fix: false]
# - gochecknoinits # Checks that no init functions are present in Go code [fast: true, auto-fix: false]
# - gocognit # Computes and checks the cognitive complexity of functions [fast: true, auto-fix: false]
- goconst # Finds repeated strings that could be replaced by a constant [fast: true, auto-fix: false]
- gocritic # Provides diagnostics that check for bugs, performance and style issues. [fast: false, auto-fix: false]
# - gocyclo # Computes and checks the cyclomatic complexity of functions [fast: true, auto-fix: false]
# - godot # Check if comments end in a period [fast: true, auto-fix: true]
# - godox # Tool for detection of FIXME, TODO and other comment keywords [fast: true, auto-fix: false]
# - goerr113 # Go linter to check the errors handling expressions [fast: false, auto-fix: false]
- gofmt # Gofmt checks whether code was gofmt-ed. By default this tool runs with -s option to check for code simplification [fast: true, auto-fix: true]
- gofumpt # Gofumpt checks whether code was gofumpt-ed. [fast: true, auto-fix: true]
- goheader # Checks is file header matches to pattern [fast: true, auto-fix: false]
- goimports # Check import statements are formatted according to the 'goimport' command. Reformat imports in autofix mode. [fast: true, auto-fix: true]
# - golint # [deprecated] # Golint differs from gofmt. Gofmt reformats Go source code, whereas golint prints out style mistakes [fast: false, auto-fix: false]
# - gomnd # An analyzer to detect magic numbers. [fast: true, auto-fix: false]
- gomoddirectives # Manage the use of 'replace', 'retract', and 'excludes' directives in go.mod. [fast: true, auto-fix: false]
- gomodguard # Allow and block list linter for direct Go module dependencies. This is different from depguard where there are different block types for example version constraints and module recommendations. [fast: true, auto-fix: false]
- goprintffuncname # Checks that printf-like functions are named with `f` at the end [fast: true, auto-fix: false]
# - gosec # (gas) # Inspects source code for security problems [fast: false, auto-fix: false]
- gosmopolitan # Report certain i18n/l10n anti-patterns in your Go codebase [fast: false, auto-fix: false]
- grouper # An analyzer to analyze expression groups. [fast: true, auto-fix: false]
# - ifshort # [deprecated] # Checks that your code uses short syntax for if-statements whenever possible [fast: true, auto-fix: false]
- importas # Enforces consistent import aliases [fast: false, auto-fix: false]
- interfacebloat # A linter that checks the number of methods inside an interface. [fast: true, auto-fix: false]
# - interfacer # [deprecated] # Linter that suggests narrower interface types [fast: false, auto-fix: false]
# - ireturn # Accept Interfaces, Return Concrete Types [fast: false, auto-fix: false]
# - lll # Reports long lines [fast: true, auto-fix: false]
- loggercheck # (logrlint) # Checks key value pairs for common logger libraries (kitlog,klog,logr,zap). [fast: false, auto-fix: false]
- maintidx # maintidx measures the maintainability index of each function. [fast: true, auto-fix: false]
- makezero # Finds slice declarations with non-zero initial length [fast: false, auto-fix: false]
# - maligned # [deprecated] # Tool to detect Go structs that would take less memory if their fields were sorted [fast: false, auto-fix: false]
- mirror # reports wrong mirror patterns of bytes/strings usage [fast: false, auto-fix: false]
- misspell # Finds commonly misspelled English words in comments [fast: true, auto-fix: true]
- musttag # enforce field tags in (un)marshaled structs [fast: false, auto-fix: false]
# - nakedret # Finds naked returns in functions greater than a specified function length [fast: true, auto-fix: false]
- nestif # Reports deeply nested if statements [fast: true, auto-fix: false]
- nilerr # Finds the code that returns nil even if it checks that the error is not nil. [fast: false, auto-fix: false]
- nilnil # Checks that there is no simultaneous return of `nil` error and an invalid value. [fast: false, auto-fix: false]
- nlreturn # nlreturn checks for a new line before return and branch statements to increase code clarity [fast: true, auto-fix: false]
- noctx # noctx finds sending http request without context.Context [fast: false, auto-fix: false]
- nolintlint # Reports ill-formed or insufficient nolint directives [fast: true, auto-fix: false]
# - nonamedreturns # Reports all named returns [fast: false, auto-fix: false]
# - nosnakecase # [deprecated] # nosnakecase is a linter that detects snake case of variable naming and function name. [fast: true, auto-fix: false]
- nosprintfhostport # Checks for misuse of Sprintf to construct a host with port in a URL. [fast: true, auto-fix: false]
- paralleltest # paralleltest detects missing usage of t.Parallel() method in your Go test [fast: false, auto-fix: false]
- prealloc # Finds slice declarations that could potentially be pre-allocated [fast: true, auto-fix: false]
- predeclared # find code that shadows one of Go's predeclared identifiers [fast: true, auto-fix: false]
# - promlinter # Check Prometheus metrics naming via promlint [fast: true, auto-fix: false]
- reassign # Checks that package variables are not reassigned [fast: false, auto-fix: false]
- revive # Fast, configurable, extensible, flexible, and beautiful linter for Go. Drop-in replacement of golint. [fast: false, auto-fix: false]
- rowserrcheck # checks whether Err of rows is checked successfully [fast: false, auto-fix: false]
# - scopelint # [deprecated] # Scopelint checks for unpinned variables in go programs [fast: true, auto-fix: false]
- sqlclosecheck # Checks that sql.Rows and sql.Stmt are closed. [fast: false, auto-fix: false]
# - structcheck # [deprecated] # Finds unused struct fields [fast: false, auto-fix: false]
- stylecheck # Stylecheck is a replacement for golint [fast: false, auto-fix: false]
- tagalign # check that struct tags are well aligned [fast: true, auto-fix: true]
# - tagliatelle # Checks the struct tags. [fast: true, auto-fix: false]
- tenv # tenv is analyzer that detects using os.Setenv instead of t.Setenv since Go1.17 [fast: false, auto-fix: false]
- testableexamples # linter checks if examples are testable (have an expected output) [fast: true, auto-fix: false]
- testpackage # linter that makes you use a separate _test package [fast: true, auto-fix: false]
- thelper # thelper detects Go test helpers without t.Helper() call and checks the consistency of test helpers [fast: false, auto-fix: false]
- tparallel # tparallel detects inappropriate usage of t.Parallel() method in your Go test codes [fast: false, auto-fix: false]
- unconvert # Remove unnecessary type conversions [fast: false, auto-fix: false]
- unparam # Reports unused function parameters [fast: false, auto-fix: false]
- usestdlibvars # A linter that detect the possibility to use variables/constants from the Go standard library. [fast: true, auto-fix: false]
# - varcheck # [deprecated] # Finds unused global variables and constants [fast: false, auto-fix: false]
# - varnamelen # checks that the length of a variable's name matches its scope [fast: false, auto-fix: false]
- wastedassign # wastedassign finds wasted assignment statements. [fast: false, auto-fix: false]
- whitespace # Tool for detection of leading and trailing whitespace [fast: true, auto-fix: true]
# - wrapcheck # Checks that errors returned from external packages are wrapped [fast: false, auto-fix: false]
- wsl # Whitespace Linter - Forces you to use empty lines! [fast: true, auto-fix: false]
- zerologlint # Detects the wrong usage of `zerolog` that a user forgets to dispatch with `Send` or `Msg`. [fast: false, auto-fix: false]

linters-settings:
errcheck:
check-type-assertions: true
goconst:
min-len: 2
min-occurrences: 3
gocritic:
enabled-tags:
- style
- diagnostic
- performance
- experimental
- style
- opinionated
disabled-checks:
- captLocal
- octalLiteral
- whyNoLint
gocyclo:
# Minimal code complexity to report.
# Default: 30 (but we recommend 10-20)
min-complexity: 10
govet:
check-shadowing: true
disabled-checks:
- fieldalignment
nolintlint:
require-explanation: true
require-specific: true
varnamelen:
# The minimum length of a variable's name that is considered "long".
# Variable names that are at least this long will be ignored.
# Default: 3
min-name-length: 2
# Check method receivers.
# Default: false
check-receiver: true
# Check named return values.
# Default: false
check-return: true
# Check type parameters.
# Default: false
check-type-param: true
whitespace:
# Enforces newlines (or comments) after every multi-line if statement.
# Default: false
multi-if: true
# Enforces newlines (or comments) after every multi-line function signature.
# Default: false
multi-func: true
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ test:

.PHONY: build
build:
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o bin/xsubfind3r cmd/xsubfind3r/main.go
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o bin/xurlfind3r cmd/xurlfind3r/main.go

.PHONY: install
install:
Expand Down
55 changes: 27 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,21 +22,12 @@

## Features

* [x] Fetches URLs from curated passive sources to maximize results.
<details>
<summary>Sources: Click to expand!</summary>

| Technique | Source |
| :-------- | :----- |
| APIs | AlienVault's OTX, BeVigil, GitHub, IntelX, URLScan |
| Web Archives | CommonCrawl, Wayback |

</details>
* [x] Parses URLs from wayback webpages and `robots.txt` snapshots.
* [x] Filters out duplicate URLs.
* [x] Supports URLs matching and filtering.
* [x] Supports `stdin` and `stdout` for easy integration into workflows.
* [x] Cross-Platform (Windows, Linux & macOS).
* Fetches URLs from curated passive sources to maximize results.
* Parses URLs from wayback webpages and `robots.txt` snapshots.
* Filters out duplicate URLs.
* Supports URLs matching and filtering.
* Supports `stdin` and `stdout` for easy integration into workflows.
* Cross-Platform (Windows, Linux & macOS).

## Installation

Expand Down Expand Up @@ -155,28 +146,38 @@ xurlfind3r -h
help message:

```
_ __ _ _ _____
__ ___ _ _ __| |/ _(_)_ __ __| |___ / _ __
_ __ _ _ _____
__ ___ _ _ __| |/ _(_)_ __ __| |___ / _ __
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
> <| |_| | | | | _| | | | | (_| |___) | |
/_/\_\\__,_|_| |_|_| |_|_| |_|\__,_|____/|_| v0.3.0
/_/\_\\__,_|_| |_|_| |_|_| |_|\__,_|____/|_|
v0.3.0

with <3 by Hueristiq Open Source

USAGE:
xurlfind3r [OPTIONS]

CONFIGURATION:
-c, --configuration string configuration file path (default: $HOME/.config/xurlfind3r/config.yaml)

INPUT:
-d, --domain string[] target domains
-d, --domain string[] target domain
-l, --list string target domains' list file path

TIP: For multiple input domains use comma(,) separated value with `-d`,
specify multiple `-d`, load from file with `-l` or load from stdin.

SCOPE:
--include-subdomains bool match subdomain's URLs

SOURCES:
--sources bool list supported sources
-u, --use-sources string[] comma(,) separated sources to use
-e, --exclude-sources string[] comma(,) separated sources to exclude
--parse-wayback-robots bool with wayback, parse robots.txt snapshots
--parse-wayback-source bool with wayback, parse source code snapshots
--sources bool list supported sources
-u, --use-sources string[] comma(,) separated sources to use
-e, --exclude-sources string[] comma(,) separated sources to exclude
--parse-wayback-robots bool with wayback, parse robots.txt snapshots
--parse-wayback-source bool with wayback, parse source code snapshots

FILTER & MATCH:
-f, --filter string regex to filter URLs
Expand All @@ -186,10 +187,8 @@ OUTPUT:
--no-color bool disable colored output
-o, --output string output URLs file path
-O, --output-directory string output URLs directory path
-v, --verbosity string debug, info, warning, error, fatal or silent (default: info)

CONFIGURATION:
-c, --configuration string configuration file path (default: ~/.hueristiq/xurlfind3r/config.yaml)
-s, --silent bool display output subdomains only
-v, --verbose bool display verbose output
```

### Examples
Expand Down
Loading

0 comments on commit c4266b7

Please sign in to comment.