全方位扫描工具，具备高效的机器探活，端口探活，协议识别，指纹识别，漏洞扫描等功能，高效爬虫，自动寻找漏洞agent，包含网站测试Agent、漏洞验证Agent(越权测试演示)、报告生成Agent

全方位扫描工具，具备高效的机器探活，端口探活，协议识别，指纹识别，漏洞扫描等功能，高效爬虫，自动寻找漏洞agent，包含网站测试Agent、漏洞验证Agent(越权测试演示)、报告生成Agent。

本工具仅供安全测试人员运用于授权测试, 禁止用于未授权测试, 违者责任自负

0x01 简介

一款全方位扫描工具，具备高效的机器探活，端口探活，协议识别，指纹识别，漏洞扫描等功能,

• 纯go实现的协议识别
• 丰富的扫描模式
  • 支持端口扫描的乱序 (目标越大，速度越快，准确度越高)
• release默认编译nuclei的8000+漏洞, 支持自定义poc
• web指纹25000+
• 优雅的html报告导出

0x02 主要功能

1. 信息收集

• 基于ICMP的主机探活,能够快速识别存活的主机
• 25000+的web指纹应用识别
• 支持web识别后的目录扫描
• 丰富的协议识别
  • mysql,redis,smb,ldap,ssh,vnc,ftp,smtp, rdp

2. 协议爆破

• 常用协议爆破 ssh,mysql,redis,vnc

3. 漏洞检测

• redis未授权
• 永恒之蓝/永恒之黑
• 内置nuclei漏洞引擎
• 自定义poc预加载目录或文件

4. 辅助功能

• 实时打印日志
• 自定义指纹文件加载
• http/socks5代理
• 支持fofa语句,目标从fofa拉取
• 支撑目标从fofa拉取目标

0x03 使用说明

目标配置

-t  指定目标(192.168.1.1/24, 192.168.1.1-192.168.1.128)
-u  指定url目标(http://www.baidu.***)
-f  从文件导入目标
-k  从fofa导入目标(-k 'domain=fofa.info')

端口配置

-p  指定端口(-p 22,80,3306 或者 -p 1-8080 or -p small)
  small:   常用前12个端口
  sql:     常用数据库端口
  all:     全端口
  ***pany: 公司常用87个端口

代理配置

--socks5  指定socks5代理(如: socks5://127.0.0.1:1080)
--http-proxy  指定http代理(如: http://127.0.0.1:1080)

扫描模式（信息收集）

-c  设置并发量
-r  乱序扫描(扫描大量目标时, 推荐使用)
-k  设置fofa key
-n  跳过icmp扫描即ping
-a  跳过目录扫描
-v  打印识别出的协议内容
-l  协议全量识别(比如mysql只识别3306, 开启后每个协议会识别所有端口)
-w  自定义web指纹加载(默认使用内置web指纹, 格式文件请参考/static/finger_new.json)
-b  配置目录扫描的文件

扫描模式（漏洞扫描）

-x         禁用漏洞扫描
-m         不进行指纹匹配,对每个存活进行全量漏洞扫描
--poc-file 自定义poc文件/目录
--poc-tags 指定poc标签
--poc-id   指定poc id

0x04 演示案例

1. 利用fofa进行外网全端口打点,并且使用乱序扫描

   milkyway.exe --fofa-query 'domain=fofa.info||host=fofa.info' -p all --no-ping --scan-random

2. 设置1000并发量使用无序扫描内网所有端口

   milkyway.exe -t 192.168.1.0/24 -p all --scan-random -c 1000 --no-ping

0x05 参数使用

milkyway.exe -t 192.168.1.1/24 -p ***pany (使用公司常用87个端口)

milkyway.exe -t 192.168.1.1/24 -p small --full-scan (对前12个端口进行全协议识别)

milkyway.exe -t 192.168.1.1/24 --no-ping (跳过icmp扫描)

milkyway.exe -t 192.168.1.1/24 --finger-file ./your_file (自定义指纹文件)

milkyway.exe -t 192.168.1.1/24 --verbose (打印协议详细信息)

milkyway.exe -t 192.168.1.1/24 --no-match (漏洞扫描不进行指纹匹配,即下发全量)

milkyway.exe -t 192.168.1.1/24 --poc-file ./your_file (自定义漏洞目录)

milkyway.exe -t 192.168.1.1/24 --poc-tags cve,***vd (指定多个poc标签)

sql: 常用数据库端口, small: 常用前12个端口, all: 全端口

0x06 运行截图

参考项目

https://github.***/shadow1ng/fscan

https://github.***/EdgeSecurityTeam/EHole

https://github.***/chainreactors/neutron

---

---

gogo工具

blog posts:

• https://chainreactors.github.io/wiki/blog/2022/11/15/gogo-introduce/

Features

• 自由的端口配置
• 支持主动/被动指纹识别
• 关键信息提取, 如title, cert 以及自定义提取信息的正则
• 支持nuclei poc, 引擎: https://github.***/chainreactors/neutron
• 无害的扫描, 每个添加的poc都经过人工审核
• 可控的启发式扫描
• 超强的性能, 最快的速度, 尽可能小的内存与CPU占用.
• 最小发包原则, 尽可能少地发包获取最多的信息
• 支持DSL, 可以通过修改的配置文件自定义自己的gogo
• 完善的输出与输出设计
• 几乎不依赖第三方库, 纯原生go编写, 在windows 2003上也可以使用完整的漏洞/指纹识别功能

QuickStart

完整的文档与教程位于wiki: https://chainreactors.github.io/wiki/gogo/

指纹与poc仓库: https://github.***/chainreactors/templates

最简使用

指定网段进行默认扫描, 并在命令行输出

gogo -i 192.168.1.1/24 -p win,db,top2

端口配置

一些常用的端口配置:

• -p - 等于-p 1-65535
• -p 1-1000 端口范围
• -p ***mon tag: ***mon 表示内网常用端口
• -p top2,top3 可以同时选择多个tag. 外网常见web端口
• -p all 表示所有预设的tag的合集.

通过逗号分割多个配置, 可根据场景进行各种各样的组合配置. 例如:

gogo -i 1.1.1.1/24 -p 1-1000,***mon,http,db

查看全部端口配置

gogo -P port

可查看所有的tag对应的端口.

当前已有端口配置: (根据端口类型分类)
         top1 :  80,443,8080
         top2 :  70,80,81,82,83,84,85,86,87,88,89,90,443,1080,2000,2001,3000,3001,1443,4443,4430,5000,5001,5601,6000,6001,6002,6003,7000,7001,7002,7003,9000,9001,9002,9003,8080,8081,8082,8083,8084,8085,8086,8087,8088,8089,8090,8091,8000,8001,8002,8003,8004,8005,8006,8007,8008,8009,8010,8011,8012,8013,8014,8015,8016,8017,8018,8019,8020,8820,6443,8443,9443,8787,7080,8070,7070,7443,9080,9081,9082,9083,5555,6666,7777,7788,9999,6868,8888,8878,8889,7890,5678,6789,9090,9091,9092,9093,9094,9095,9096,9097,9098,9099,9100,9988,9876,8765,8099,8763,8848,8161,8060,8899,800,801,888,10000,10001,10002,10003,10004,10005,10006,10007,10008,10009,10010,1081,1082,10080,10443,18080,18000,18088,18090,19090,19091,50070
         top3 :  444,9443,6080,6443,9070,9092,9093,7003,7004,7005,7006,7007,7008,7009,7010,7011,9003,9004,9005,9006,9007,9008,9009,9010,9011,8100,8101,8102,8103,8104,8105,8106,8107,8108,8109,8110,8111,8161,8021,8022,8023,8024,8025,8026,8027,8028,8029,8030,8880,8881,8882,8883,8884,8885,8886,8887,8888,8889,8890,8010,8011,8012,8013,8014,8015,8016,8017,8018,8019,8020,8090,8091,8092,8093,8094,8095,8096,8097,8098,8099,8180,8181,8983,1311,8363,8800,8761,8873,8866,8900,8282,8999,8989,8066,8200,8040,8060,10800,18081
         docker :  2375,2376,2377,2378,2379,2380
         lotus :  1352
         dubbo :  18086,20880,20881,20882
         oracle :  1158,1521,11521,210
         ...
         ...
         ...

启发式扫描

当目标范围的子网掩码小于24时, 建议启用 smart模式扫描(原理见doc), 例如子网掩码为16时(输出结果较多, 建议开启–af输出到文件, 命令行只输出日志)

gogo -i 172.16.1.1/12 -m ss --ping -p top2,win,db --af

--af 表示自动指定文件生成的文件名.

-m ss 表示使用supersmart模式进行扫描. 还有ss,sc模式适用于不同场景

--ping 表示在指纹识别/信息获取前判断ip是否能被ping通, 减少无效发包. 需要注意的是, 不能被ping通不代表目标一定不存活, 使用时请注意到这一点

workflow

启发式扫描的命令有些复杂, 但可以使用workflow将复杂的命令写成配置文件, 快捷调用(内置的workflow细节见doc).

gogo -w 172

即可实现与gogo -i 172.16.1.1/12 -m ss --ping -p top2,win,db --af 完全相同的配置

查看所有workflow

gogo -P workflow

常用的配置已经被集成到workflow中, 例如使用supersmart mod 扫描10段内网, gogo -w 10即可.

还有一些预留配置(即填写了其他配置, 但没有填写目标, 需要-i手动指定目标), 例如:

gogo -w ss -i 11.0.0.0/8

workflow中的预设参数优先级低于命令行输入, 因此可以通过命令行覆盖workflow中的参数. 例如:

gogo -w 10 -i 11.0.0.0/8

示例

一个简单的任务

gogo -i 81.68.175.32/28 -p top2

gogo -i 81.68.175.32/28 -p top2
[*] Current goroutines: 1000, Version Level: 0,Exploit Target: none, PortSpray: false ,2022-07-07 07:07.07
[*] Start task 81.68.175.32/28 ,total ports: 100 , mod: default ,2022-07-07 07:07.07
[*] ports: 80,81,82,83,84,85,86,87,88,89,90,443,1080,2000,2001,3000,3001,4443,4430,5000,5001,5601,6000,6001,6002,6003,7000,7001,7002,7003,9000,9001,9002,9003,8080,8081,8082,8083,8084,8085,8086,8087,8088,8089,8090,8000,8001,8002,8003,8004,8005,8006,8007,8008,8009,8010,8011,8012,8013,8014,8015,8016,8017,8018,8019,8020,6443,8443,9443,8787,7080,8070,7070,7443,9080,9081,9082,9083,5555,6666,7777,9999,6868,8888,8889,9090,9091,8091,8099,8763,8848,8161,8060,8899,800,801,888,10000,10001,10080 ,2022-07-07 07:07.07
[*] Scan task time is about 8 seconds ,2022-07-07 07:07.07
[+] http://81.68.175.33:80      nginx/1.16.0            nginx                   bd37 [200] HTTP/1.1 200
[+] http://81.68.175.32:80      nginx/1.18.0 (Ubuntu)           nginx                   8849 [200] Wel***e to nginx!
[+] http://81.68.175.34:80      nginx           宝塔||nginx                     f0fa [200] 没有找到站点
[+] http://81.68.175.34:8888    nginx           nginx                   d41d [403] HTTP/1.1 403
[+] http://81.68.175.34:3001    nginx           webpack||nginx                  4a9b [200] shop_mall
[+] http://81.68.175.37:80      Microsoft-IIS/10.0              iis10                   c80f [200] HTTP/1.1 200             c0f6 [200] 安全入口校验失败
[*] Alive sum: 5, Target sum : 1594 ,2022-07-07 07:07.07
[*] Totally run: 4.0441884s ,2022-07-07 07:07.07

如果要联动其他工具, 可以指定-q/--quiet关闭日志信息, 只保留输出结果.

输出与再处理

关于输入输出以及各种高级用法请见output的wiki

如果执行gogo -i 81.68.175.1 --af

扫描完成后, 可以看到在gogo二进制文件同目录下, 生成了.81.68.175.1_28_all_default_json.dat1, 该文件是deflate压缩的json文件.

通过gogo格式化该文件, 获得human-like的结果

 gogo  -F .\.81.68.175.1_28_all_default_json.dat1
Scan Target: 81.68.175.1/28, Ports: all, Mod: default
Exploit: none, Version level: 0

[+] 81.68.175.32
        http://81.68.175.32:80  nginx/1.18.0 (Ubuntu)           nginx                   8849 [200] Wel***e to nginx!
        tcp://81.68.175.32:22                   *ssh                     [tcp]
        tcp://81.68.175.32:389                                           [tcp]
[+] 81.68.175.33
        tcp://81.68.175.33:3306                 *mysql                   [tcp]
        tcp://81.68.175.33:22                   *ssh                     [tcp]
        http://81.68.175.33:80  nginx/1.16.0            nginx                   bd37 [200] HTTP/1.1 200
[+] 81.68.175.34
        tcp://81.68.175.34:3306                 mysql 5.6.50-log                         [tcp]
        tcp://81.68.175.34:21                   ftp                      [tcp]
        tcp://81.68.175.34:22                   *ssh                     [tcp]
        http://81.68.175.34:80  nginx           宝塔||nginx                     f0fa [200] 没有找到站点
        http://81.68.175.34:8888        nginx           nginx                   d41d [403] HTTP/1.1 403
        http://81.68.175.34:3001        nginx           webpack||nginx                  4a9b [200] shop_mall
[+] 81.68.175.35
        http://81.68.175.35:47001       Microsoft-HTTPAPI/2.0           microsoft-httpapi                       e702 [404] Not Found
[+] 81.68.175.36
        http://81.68.175.36:80  nginx   PHP     nginx                   babe [200] 风闻客栈24小时发卡中心 - 风闻客栈24小时发卡中心
        tcp://81.68.175.36:22                   *ssh                     [tcp]
...
...

导出到其他工具

一些常用的输出格式.

• -o full 默认输出格式, 即上面示例所示.
• -o color 带颜色的full输出. 在v2.11.0版本之后, -F 输出到命令行时为默认开启状态. 如果需要关闭, 手动指定-o full即可
• -o jl 一行一个json, 可以通过管道传给jq实时处理
• -o json 一个大的json文件
• -o url 只输出url, 通常在-F时使用

所有的输出格式见: https://chainreactors.github.io/wiki/gogo/start/#_4

输出过滤器

--filter 参数可以从dat文件中过滤出指定的数据并输出.

例如过滤指定字段的值: gogo -F 1.dat --filter framework::redis -o target 表示从1.dat中过滤出redis的目标, 并输出为target字段.

其中:: 表示模糊匹配, 还有其他三种语法,如 == 为精准匹配, != 为不等于, !: 为不包含

-F 1.json -f file 重新输出到文件, 也可以-F 1.dat --af 自动生成格式化后的文件名.

注意事项

• (重要)因为并发过高,可能对路由交换设备造成伤害, 例如某些家用路由设备面对高并发可能会死机, 重启, 过热等后果. 因此在外网扫描的场景下建议在阿里云,华为云等vps上使用,如果扫描国外资产,建议在国外vps上使用.本地使用如果网络设备性能不佳会带来大量丢包. 如果在内网扫描需要根据实际情况调整并发数.
• 如果使用中发现疯狂报错,大概率是io问题(例如多次扫描后io没有被正确释放,或者配合proxifier以及类似代理工具使用报错),可以通过重启电脑,或者虚拟机中使用,关闭代理工具解决.如果依旧无法解决请联系我们.
• 还需要注意,upx压缩后的版本虽然体积小,但是有可能被杀软杀,也有可能在部分机器上无法运行.
• 一般情况下无法在代理环境中使用,除非使用-t参数指定较低的速率(默认并发为4000).
• gogo本身并不具备任何攻击性, 也无法对任何漏洞进行利用.
• 使用gogo需先确保获得了授权, gogo反对一切非法的黑客行为

使用场景并发推荐

默认的并发linux为4000, windows为1000, 为企业级网络环境下可用的并发. 不然弱网络环境(家庭, 基站等)可能会导致网络dos

建议根据不同环境,手动使用-t参数指定并发数.

• 家用路由器(例如钓鱼, 物理, 本机扫描)时, 建议并发 100-500
• linux 生产网网络环境(例如外网突破web获取的点), 默认并发4000, 不需要手动修改
• windows 生产网网络环境, 默认并发1000, 不需要手动修改
• 高并发下udp协议漏报较多, 例如获取***bois信息时, 建议单独对udp协议以较低并发重新探测
• web的正向代理(例如regeorg),建议并发 10-30
• 反向代理(例如frp), 建议并发10-100

如果如果发生大量漏报的情况, 大概率是网络环境发生的阻塞, 倒是网络延迟上升超过上限.

因此也可以通过指定 -d 5 (tcp默认为2s, tls默认为两倍tcp超时时间,即4s)来提高超时时间, 减少漏报.

未来也许会实现auto-tune, 自动调整并发速率

这些用法大概只覆盖了一小半的使用场景, 请阅读文档

Make

手动编译

# download
git clone --recurse-submodules https://github.***/chainreactors/gogo
cd gogo/v2

# sync dependency
go mod tidy   

# generate template.go
# 注意: 如果需要使用go1.10编译windows03可用版本， 也需要先使用高版本的go generate生成相关依赖
go generate  

# build 
go build .

# windows server 2003 ***pile
GOOS=windows GOARCH=386 go1.10 build .

# 因为go1.10 还没有go mod, 可能会导致依赖报错. 如果发生了依赖报错, 可以使用go1.11 编译. 
# go1.11 官方声明不支持windows server 2003 , 实测可以稳定运行(需要调低并发).
GOOS=windows GOARCH=386 go1.11 build .

如果需要编译windows xp/2003的版本, 请先使用高版本的go生成templates. 再使用go 1.11编译即可.

Similar or related works

• ServerScan 早期的简易扫描器, 功能简单但开拓了思路
• fscan 简单粗暴的扫描器, 细节上有不少问题, 但胜在简单. 参考其直白的命令行，设计了workflow相关功能.
• kscan 功能全面的扫描器, 从中选取合并了部分指纹
• ladongo 集成了各种常用功能, 从中学习了多个特殊端口的信息收集
• cube 与fscan类似, 从中学习了NTLM相关协议的信息收集

gogo从这些相似的工作中改进自身. 感谢前人的工作.

细节上的对比请看文档

THANKS

• https://github.***/projectdiscovery/nuclei-templates
• https://github.***/projectdiscovery/nuclei

---

---

xnLinkFinder 高效爬虫工具

This is a tool used to discover endpoints (and potential parameters) for a given target. It can find them by:

• crawling a target (pass a domain/URL)
• crawling multiple targets (pass a file of domains/URLs)
• searching files in a given directory (pass a directory name)
• search a single file’s contents
• get them from a Burp project (pass location of a Burp XML file)
• get them from an ZAP project (pass location of a ZAP ASCII message file)
• get them from a Caido project (pass location of a Caido export CSV file)
• processing a waymore results directory (searching archived response files from waymore -mode R and also requesting URLs from waymore.txt and the original URLs from waymore_index.txt - see waymore README.md)

The python script is based on the link finding capabilities of my Burp extension GAP.
As a starting point, I took the amazing tool LinkFinder by Gerben Javado, and used the Regex for finding links, but with additional improvements to find even more.

Installation

xnLinkFinder supports Python 3.

Install xnLinkFinder in default (global) python environment.

pip install xnLinkFinder

OR

pip install git+https://github.***/xnl-h4ck3r/xnLinkFinder.git -v

You can upgrade with

pip install --upgrade xnLinkFinder

pipx

Quick setup in isolated python environment using pipx

pipx install git+https://github.***/xnl-h4ck3r/xnLinkFinder.git

Usage

Arg	Long Arg	Description
-i	–input	Input a: URL, text file of URLs, a Directory of files to search, a Burp XML output file, a ZAP output file, or a Caido CSV, or a single file to search contents.
-o	–output	The file to save the Links output to, including path if necessary (default: output.txt). If set to cli then output is only written to STDOUT. If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-op	–output-params	The file to save the Potential Parameters output to, including path if necessary (default: parameters.txt). If set to cli then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-owl	–output-wordlist	The file to save the target specific Wordlist output to, including path if necessary (default: No wordlist output). If set to cli then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-oo	–output-oos	The file to save the out of scope links output to, including path if necessary (default: No OOS output). If set to cli then output is only written to STDOUT (but not piped to another program). If the file already exist it will just be appended to (and de-duplicated) unless option -ow is passed.
-ow	–output-overwrite	If the output file already exists, it will be overwritten instead of being appended to.
-sp	–scope-prefix	Any links found starting with / will be prefixed with scope domains in the output instead of the original link. If the passed value is a valid file name, that file will be used, otherwise the string literal will be used.
-spo	–scope-prefix-original	If argument -sp is passed, then this determines whether the original link starting with / is also included in the output (default: false)
-spkf	–scope-prefix-keep-failed	If argument -spkf is passed, then this determines whether a prefixed link will be kept in the output if it was a 404 or a RequestException o***urs (default: false)
-sf	–scope-filter	Will filter output links to only include them if the domain of the link is in the scope specified. If the passed value is a valid file name, that file will be used, otherwise the string literal will be used. This argument is now mandatory if input is a domain/URL (or file of domains/URLs) to prevent crawling sites that are not in scope and also preventing misleading results.
-c	–cookies †	Add cookies to pass with HTTP requests. Pass in the format 'name1=value1; name2=value2;'
-H	–headers †	Add custom headers to pass with HTTP requests. Pass in the format 'Header1: value1; Header2: value2;'
-ra	–regex-after	RegEx for filtering purposes against found endpoints before output (e.g. /api/v[0-9]\.[0-9]\* ). If it matches, the link is output.
-d	–depth †	The level of depth to search. For example, if a value of 2 is passed, then all links initially found will then be searched for more links (default: 1). This option is ignored for Burp files, ZAP files and Caido files because they can be huge and consume lots of memory. It is also advisable to use the -sp (--scope-prefix) argument to ensure a request to links found without a domain can be attempted.
-p	–processes †	Basic multithreading is done when getting requests for a URL, or file of URLs (not a Burp file, ZAP file or Caido file). This argument determines the number of processes (threads) used (default: 25)
-x	–exclude	Additional Link exclusions (to the list in config.yml) in a ***ma separated list, e.g. careers,forum
-orig	–origin	Whether you want the origin of the link to be in the output. Displayed as LINK-URL [ORIGIN-URL] in the output (default: false)
-prefixed		Whether you want to see which links were prefixed in the output. Displays (PREFIXED) after link and origin in the output (default: false)
-xrel	–exclude-relative-links	By default, if any links in the results start with ./ or ../, they will be included. If this argument is used, these relative links will not be added.
-t	–timeout †	How many seconds to wait for the server to send data before giving up (default: 10 seconds)
-inc	–include	Include input (-i) links in the output (default: false)
-u	–user-agent †	What User Agents to get links for, e.g. -u desktop mobile. Possible values are desktop, mobile, set-top-boxes and game-console. Also there are mobile-apple, mobile-android and mobile-windows that are subsets of mobile but can be used separately.
-uc	–user-agent-custom †	A custom User Agent string to use for all requests. This will override the -u/--user-agent argument. This can be used when a program requires a specific User Agent header to identify you for example.
-insecure	†	Whether TLS certificate checks should be disabled when making requests (delfault: false)
-s429	†	Stop when > 95 percent of responses return 429 Too Many Requests (default: false)
-s403	†	Stop when > 95 percent of responses return 403 Forbidden (default: false)
-sTO	†	Stop when > 95 percent of requests time out (default: false)
-sCE	†	Stop when > 95 percent of requests have connection errors (default: false)
-m	–memory-threshold	The memory threshold percentage. If the machines memory goes above the threshold, the program will be stopped and ended gracefully before running out of memory (default: 95)
-mfs	–max-file-size †	The maximum file size (in bytes) of a file to be checked if -i is a directory. If the file size is over, it will be ignored (default: 500 MB). Setting to 0 means no files will be ignored, regardless of size.
-rp	–replay-proxy†	For active link finding with URL (or file of URLs), replay the requests through this proxy.
-ascii-only		Whether links and parameters will only be added if they only contain ASCII characters. This can be useful when you know the target is likely to use ASCII characters and you also get a number of false positives from binary files for some reason.
-mtl	–max-time-limit	The maximum time limit (in minutes) to run before stopping (default: 0). If 0 is passed, there is no limit.
	–config	Path to the YML config file. If not passed, it looks for file config.yml in the default directory, typically ~/.config/xnLinkFinder.
-nwlpl	–no-wordlist-plurals	When words are found for a target specific wordlist, by default new words are added if there is a singular word from a plural, and vice versa. If this argument is used, this process is not done.
-nwlpw	–no-wordlist-pathwords	By default, any path words found in the links will be processed for the target specific wordlist. If this argument is used, they will not be processed. NOTE: if the YAML config value of respParamPathWords is True then this argument will not have any effect unless -nwlpm/--no-wordlist-parameters is also passed.
-nwlpm	–no-wordlist-parameters	By default, any parameters found in the links will be processed for the target specific wordlist. If this argument is used, they will not be processed.
-nwlc	–no-wordlist-***ments	By default, any ***ments in pages will be processed for the target specific wordlist. If this argument is used, they will not be processed.
-nwlia	–no-wordlist-imgalt	By default, any image ‘alt’ attributes will be processed for the target specific wordlist. If this argument is used, they will not be processed.
-nwld	–no-wordlist-digits	Exclude any words from the target specific wordlist with numerical digits in.
-nwll	–no-wordlist-lowercase	By default, any word added with any uppercase characters in will also add the word in lowercase. If this argument is used, the lowercase words will not be added.
-wlml	–wordlist-maxlen	The maximum length of words to add to the target specific wordlist, excluding plurals (default: 0 - no limit)
-swf	–stopwords-file	A file of additional Stop Words (in addition to “stopWords” in the YML Config file) used to exclude words from the target specific wordlist. Stop Words are used in Natural Language Processing and different lists can be found in different libraries. You may want to add words in different languages, depending on your target.
-brt	–burpfile-remove-tags	When the input passed with -i is a Burp file, the user is asked interactively whether they want to remove unnecessary tags from that file (sometimes there is a problem in Burp XML files that can often be resolved by removing unnecessary tags which will also make the file smaller). If you are using xnLinkFinder in a script, you don’t want to break for user input, so you can set that by passing this argument with a true or false. NOTE: This is a permanent change to the file
-all	–all-tlds	All links found will be returned, even if the TLD is not ***mon. This can result in a number of false positives where variable names, etc. may also be a possible genuine domain. By default, only links that have a TLD in the ***mon TLDs (***monTLDs in config.yml) will be returned.
-cl	–content-length	Show the Content-Length of the response when crawling.
-nb	–no-banner	Hides the tool banner.
-v	–verbose	Verbose output
-vv	–vverbose	Increased verbose output
	–version	Show current version number.
-h	–help	show the help message and exit

† NOT RELEVANT FOR INPUT OF DIRECTORY, BURP XML FILE, ZAP FILE OR CAIDO CSV FILE

config.yml

The config.yml file (typically in ~/.config/xnLinkFinder/) has the keys which can be updated to suit your needs:

• linkExclude - A ***ma separated list of strings (e.g. .css,.jpg,.jpeg etc.) that all links are checked against. If a link includes any of the strings then it will be excluded from the output. If the input is a directory, then file names are checked against this list.
• contentExclude - A ***ma separated list of strings (e.g. text/css,image/jpeg,image/jpg etc.) that all responses Content-Type headers are checked against. Any responses with the these content types will be excluded and not checked for links.
• fileExtExclude - A ***ma separated list of strings (e.g. .zip,.gz,.tar etc.) that all files in Directory mode are checked against. If a file has one of those extensions it will not be searched for links. Also, in normal mode, if a response doesn’t have a content-type to check for exclusions, it will check for these extensions at the end of the URL to determine if to search for links.
• regexFiles - A list of file types separated by a pipe character (e.g. php|php3|php5 etc.). These are used in the Link Finding Regex when there are findings that aren’t obvious links, but are interesting file types that you want to pick out. If you add to this list, ensure you escape any dots to ensure correct regex, e.g. js\.map
• respParamLinksFound † - Whether to get potential parameters from links found in responses: True or False
• respParamPathWords † - Whether to add path words in retrieved links as potential parameters: True or False
• respParamJSON † - If the MIME type of the response contains JSON, whether to add JSON Key values as potential parameters: True or False
• respParamJSVars † - Whether javascript variables set with var, let or const are added as potential parameters: True or False
• respParamXML † - If the MIME type of the response contains XML, whether to add XML attributes values as potential parameters: True or False
• respParamInputField † - If the MIME type of the response contains HTML or JAVASCRIPT (because HTML could be built in HTML), whether to add NAME and ID attributes of any INPUT (or TEXTAREA) fields as potential parameters: True or False
• respParamMetaName † - If the MIME type of the response contains HTML, whether to add NAME attributes of any META tags as potential parameters: True or False
• wordsContentTypes - A ***ma separated list of strings (e.g. text/html,text/plain) to specify which response content types will be searched for words to go in the target specific wordlist.
• stopWords - A ***ma separated list of strings (e.g. then,this,that) to specify words that are excluded from the target specific wordlist. This default list is initially made up of English determiners, coordinating conjuctions and prepositions, plus a list of stop words from Scikit-Learn, a python machine learning library.
• ***monTLDs - A ***ma separated list of the most ***mon TLDs. Unless -all/--all-tlds argument is passed, only links with domains that have a TLD in this list are returned.

† IF THESE ARE NOT FOUND IN THE CONFIG FILE THEY WILL DEFAULT TO True

Examples

Find Links from a specific target - Basic

xnLinkFinder -i target.*** -sf target.***

Find Links from a specific target - Detailed

Ideally, provide scope prefix (-sp) with the primary domain (including schema), and a scope filter (-sf) to filter the results only to relevant domains (this can be a file or in scope domains). Also, you can pass cookies and customer headers to ensure you find links only available to authorised users.
Specifying the User Agent (-u desktop mobile) will first search for all links using desktop User Agents, and then try again using mobile user agents. There could be specific endpoints that are related to the user agent given. Giving a depth value (-d) will keep sending request to links found on the previous depth search to find more links.

xnLinkFinder -i target.*** -sp target_prefix.txt -sf target_scope.txt -spo -inc -vv -H 'Authorization: Bearer XXXXXXXXXXXXXX' -c 'SessionId=MYSESSIONID' -u desktop mobile -d 10

Find Links from a list of URLs - Basic

If you have a file of JS file URLs for example, you can look for links in those:

xnLinkFinder -i target_js.txt -sf target.***

NOTE: A passed file is assumed to be a list of URLs if the first line starts with // or http, otherwise it is considered to be a file to search the contents for (unless it is a Burp, Zap or Caido file).

Find Links from the contents of a file - Basic

If you have a saved response for example, you can look for links in those:

xnLinkFinder -i response.txt -sf target.***

NOTE: A passed file is assumed to be a list of URLs if the first line starts with // or http, otherwise it is considered to be a file to search the contents for (unless it is a Burp, Zap or Caido file).

Find Links from a files in a directory - Basic

If you have a files, e.g. JS files, HTTP responses, etc. you can look for links in those:

xnLinkFinder -i ~/.config/waymore/results/target.***

NOTE: Sub directories are also checked. The -mfs option can be specified to skip files over a certain size.

Find Links from a Burp project - Basic

In Burp, select the items you want to search by highlighting the scope for example, right clicking and selecting the Save selected items. Ensure that the option base64-encode requests and responses option is checked before saving.
To get all links from the file (even with HUGE files, you’ll be able to get all the links):

xnLinkFinder -i target_burp.xml

NOTE: xnLinkFinder makes the assumption that if the first line of the file passed with -i starts with <?xml then you are trying to process a Burp file.

Find Links from a Burp project - Detailed

Ideally, provide scope prefix (-sp) with the primary domain (including schema), and a scope filter (-sf) to filter the results only to relevant domains.

xnLinkFinder -i target_burp.xml -o target_burp.txt -sp https://www.target.*** -sf target.* -ow -spo -inc -vv

Find Links from a ZAP project - Basic

In ZAP, select the items you want to search by highlighting the History for example, clicking menu Export and selecting Export Messages to File.... This will let you save an ASCII text file of all requests and responses you want to search.
To get all links from the file (even with HUGE files, you’ll be able to get all the links):

xnLinkFinder -i target_zap.txt

NOTE: xnLinkFinder makes the assumption that if the first line of the file passed with -i is in the format ==== 99 ========== (v2.11.1) or ===99 ========== (v2.12) for example, then you are trying to process a ZAP ASCII text file.

Find Links from a Cadio export CSV file - Basic

In Caido, go to the History section and select the Export option.

If you are using Caido Pro or Enterprise edition, then choose the Export current rows option and pick As CSV. Go to the Exports section and download the CSV file. Then pass as input:

xnLinkFinder -i 2023-03-18-010332_csv_requests.csv

If you are using Caido ***munity edition, then you will have to choose the Export all option and pick As CSV. Go to the Exports section and download the CSV file. As you have the full history, you will want to remove anything that is not relevant from the CSV file. Use the example below, where redbull is the main part of the domains of the target you are looking at.

cat 2023-03-18-010332_csv_requests.csv | grep -E '^id|^[0-9]+,[^,]*redbull' > caido_redbull.csv
xnLinkFinder -i caido_redbull.csv

NOTE: xnLinkFinder makes the assumption that if the first line of the file passed with -i is in the format id,host,method, then you are trying to process a Caido export CSV file.

Find Links from a Waymore results directory

The waymore tool can be used to get URLs from various third party APIs, and also download archived responses from various sources. Passing a waymore results directory to xnLinKFinder will search the contents of archived responses, and also request URLs from waymore.txt and also the archived URLs from waymore_index.txt (or index.txt for older versions of waymore) and get more links from those responses. If -d/--depth is zero, then the URLs from waymore_index.txt will just be returned but not requested.

xnLinkFinder -i ~/Tools/waymore/results/target.***

NOTE: It is passed as a normal directory, but xnLinkFinder will determine it is a waymore results directory and process respectively. This relies on the default naming convention of the URLs file being waymore.txt and that file being in the same directory as the archived files (which it is by default).

Piping to other Tools

You can pipe xnLinkFinder to other tools. Any errors are sent to stderr and any links found are sent to stdout. The output file is still created in addition to the links being piped to the next program. However, potential parameters are not piped to the next program, but they are still written to file. For example:

xnLinkFinder -i redbull.*** -sp https://redbull.*** -sf rebbull.* -d 3 | unfurl keys | sort -u

You can also pass the input through stdin instead of -i.

cat redbull_subs.txt | xnLinkFinder -sp https://redbull.*** -sf rebbull.* -d 3

NOTE: You can’t pipe in a Burp, ZAP or Caido file, these must be passed using -i.

Re***mendations and Notes

• Always use the Scope Prefix argument -sp. This can be one scope domain, or a file containing multiple scope domains.
  Below are examples of the format used (no path should be included, and no wildcards used. Schema is optional, but will default to http):

  http://www.target.***
  https://target-payments.***
  https://static.target-cdn.***
  

  If a link is found that has no domain, e.g. /path/to/example.js then giving passing -sp http://www.target.*** will result in teh output http://www.target.***/path/to/example.js and if Depth (-d) is >1 then a request will be able to be made to that URL to search for more links. If a file of domains are passed using -sp then the output will include each domain followed by /path/to/example.js and increase the chance of finding more links.
• If you use -sp but still want the original link of /path/to/example.js (without a domain) additionally returned in the output, the pass the argument -spo.
• Always use the Scope Filter argument -sf. This will ensure that only relevant domains are returned in the output, and more importantly if Depth (-d) is >1 then out of scope targets will not be searched for links or parameters. This can be one scope domain, or a file containing multiple scope domains. Below are examples of the format used (no schema or path should be included):

  target.*
  target-payments.***
  static.target-cdn.***
  

  THIS IS FOR FILTERING THE LINKS DOMAIN ONLY.
• If you want to filter the final output in any way, use -ra. It’s always a good idea to use https://regex101.***/ to check your Regex expression is going to do what you expect.
• Use the -v option to have a better idea of what the tool is doing.
• If you have problems, use the -vv option which may show errors that are o***urring, which can possibly be resolved, or you can raise as an issue on github.
• Pass cookies (-c), headers (-H) and regex (-ra) values within single quotes, e.g. -ra '/api/v[0-9]\.[0-9]\*'
• Set the -o option to give a specific output file name for Links, rather than the default of output.txt. If you plan on running a large depth of searches, start with 2 with option -v to check what is being returned. Then you can increase the Depth, and the new output will be appended to the existing file, unless you pass -ow.
• Set the -op option to give a specific output file name for Potential Parameters, rather than the default of parameters.txt. Any output will be appended to the existing file, unless you pass -ow.
• If using a high Depth (-d) be wary of some sites using dynamic links so will it will just keep finding new ones. If no new links are being found, then xnlLinkFinder will stop searching. Providing the Stop flags (s429, s403, sTO, sCE) should also be considered.
• If you are finding a large number of links, especially if the Depth (-d value) is high, and have limited resources, the program will stop when it reaches the memory Threshold (-m) value and end gracefully with data intact before getting killed.
• If you decide to cancel xnLinkFinder (using Ctrl-C) in the middle of running, be patient and any gathered data will be saved before ending gracefully.
• Using the -orig option will show the URL where the link was found. This can mean you have duplicate links in the output if the same link was found on multiple sources, but it will suffixed with the origin URL in square brackets.
• When making requests, xnLinkFinder will use a random User-Agent from the current group, which defaults to desktop (unless the -uc/--user-agent-custom argument is used). If you have a target that could have different links for different user agent groups, then specify -u desktop mobile for example (separate with a space). The mobile user agent option is an ***bination of mobile-apple, mobile-android and mobile-windows. Possible values are desktop, mobile, set-top-boxes and game-console.
• When -i has been set to a directory, the contents of the files in the root of that directory will be searched for links. Files in sub-directories are not searched. Any files that are over the size set by -mfs (default: 500 MB) will be skipped.
• When using the -replay-proxy option, sometimes requests can take longer. If you start seeing more Request Timeout errors (you’ll see errors if you use -v or -vv options) then consider using -t to raise the timeout limit.
• If you know a target will only have ASCII characters in links and parameters then consider passing -ascii-only. This can eliminate a number of false positives that can sometimes get returned from binary data.
• If you pass a waymore results directory, it is worth passing the -d/--depth argument to search any extra links found from URL requests and also the -u/--user-agent if you think there could be different content found, e.g. -u desktop mobile.
• Always pass the -owl/--output-wordlist filename to save the target specific wordlist. This list can be very useful when fuzzing a target.
• The words for the target specific wordlist are taken from the following sources (any of 3 characters or more), but are also determined by the other wordlist arguments (see Usage section above):
  • All responses with certain conditions:
    • Only responses with content types specific in the YML config wordsContentTypes section are searched. The defaults are text/html,application/xml,application/json,text/plain
    • Words from <meta> tag content where:
      • Property is og:title or og:description
      • Name is description,keywords,twitter:title or twitter:description
    • Words from HTML ***ments
    • Words from alt attribute of <img> tags
    • Words from the rest of the inner HTML of the page, excluding tags <style>, <script> and <link>
  • Words found from path words in links found.
  • Parameters found from responses and links.
  • All valid words will also have the singular/plural version added to the wordlist if possible.
  • If the original word has any upper case characters, a lower case version will also be added
• If the default “Stop Words” for a target specific wordlist are not good enough, either change in the YML config file, or provide additional stop words using the -swf/--stopwords-file option. You may want to include stop words in another language, depending on the target. Stop words are used in Natural Language Processing (NLP) and many stop word lists can be found online to suit different needs.

Issues

If you ***e across any problems at all, or have ideas for improvements, please feel free to raise an issue on Github. If there is a problem, it will be useful if you can provide the exact ***mand you ran and a detailed description of the problem. If possible, run with -vv to reproduce the problem and let me know about any error messages that are given.

TODO

• I seem to have ***pleted all the TODO’s I originally had! If you think of any that need adding, let me know 🤘

Example output

Active link finding for a domain:

Piped input and output:

---

---

AI Agent 自动寻找漏洞Agent

本演示项目展示了 EZ AI Agent 如何应用于网站渗透测试。它利用 EZ 强大的被动扫描能力，并结合多 Agent 协作框架，以提升渗透测试效率。

核心特性 (Demo 版本)

• 底层执行层:

  • EZ 被动扫描引擎: 负责基础漏洞检测，提供可靠的漏洞发现能力。
  • Browser-Use 驱动: 模拟浏览器交互，覆盖更广泛的应用场景。
  • Steel 沙箱: 确保浏览器环境隔离，保障测试安全。

• 智能调度层:

  • DeepSeek (或其他 LLM 大模型) Orchestrator: 作为智能调度中心，实时分析扫描结果、页面内容和历史数据，动态调度测试 Agent，优化测试流程。

• 多 Agent 协作 (部分功能演示):

  • 网站测试 Agent: 模拟用户操作，覆盖业务场景，进行功能性测试。
  • 漏洞验证 Agent (越权测试演示): 基于视觉模型对比权限差异，辅助越权漏洞的验证。
  • 报告生成 Agent: 结合 EZ 工具的被动扫描结果和 Agent 渗透测试报告，自动生成全面的测试报告。
  • 更多 Agent 敬请期待… (后续版†本将包含更多专业的渗透测试 Agent)

https://github.***/user-attachments/assets/056057aa-b60e-478e-b0eb-88b6d86b1a3d

使用说明 (Demo 版本)

请按照以下步骤体验 EZ AI Agent Demo 的基本功能。

EZ_AI_AGENT
.
├── ez_ai_agent_demo
├── ez_target
├── ez_tools # EZ 自动化渗透测试工具目录
├── steel-browser
├── init.sql
├── LICENSE
├── README.md
├── docker-***pose.yaml
└── .env # 配置文件

1. 启动 EZ 被动扫描功能 (EZ Passive Scanning)

重要提示：请务必先下载 EZ 工具，并配置好证书。 这是启动 EZ 被动扫描功能的前提。

将 EZ 工具和证书放置于 ez_tools 文件夹下。

详细配置和介绍请参考 ez_tools 目录下的 README.md 文件。

cd ez_tools

./ez webscan --listen 0.0.0.0:9999 --disable-pocs web-brute --pocs beta-***mon-fileupload,php-path-disclosure,sqldet,beta-sqldet,php-realpath-leak

2. 配置环境变量(.env)

在 .env 文件中配置 AI 大模型的 api_key 和 base_url。

还需配置 EZ 工具被动代理的端口，由于被动代理需要 docker 网络访问使用，因此需要使用本机的内网地址而不是 localhost.

大模型 api 申请和注意事项

国内可以使用腾讯云或deepseek等平台申请api_key，使用其提供的 deepseek模型。

对于 browser-use 的调用，Google 的 gemini-2.0-flash 模型速度和精准度优于deepseek-chat(v3)，且提示词在gemini-2.0-flash模型上进行了优化，因此采用gemini模型会有更好的体验。

Gemini Api 申请地址(需科学上网)：https://aistudio.google.***/

国内可以使用 腾讯云 或 deepseek 平台 申请 api_key，并使用其提供的 deepseek 模型进行体验。

deepseek 模型无法开启 browser-use 的视觉模型，效果减半。(browser-use 使用 deepseek 模型时，请将 USE_VISION 设置为 False)

3. 启动靶场 (Target Environment) 和 智能体(Agent)

docker ***pose up

靶场地址 (Target Website)：http://内网IP地址:8089 (请将 “内网IP地址” 替换为您的实际内网 IP 地址)

4. 开始自动化渗透测试 (Start Pe***ration Testing)

1. 访问用户界面 (A***ess UI): 打开浏览器，访问 http://127.0.0.1:5173/
2. 输入目标地址 (Enter Target Address): 在 Logs 选项卡中，输入您的靶场地址 (例如：http://本机内网地址:8089)。
3. 开始测试 (Start Testing): 点击 “开始测试” 按钮。

5. 架构

下图展示了 EZ AI Agent 的整体架构设计：

---

参考的开源项目

• https://github.***/mainframe***puter/orchestra
• https://github.***/browser-use/browser-use/
• https://github.***/steel-dev/steel-browser
• …