warmer

A HTTP load testing and CDN cache warming tool in Rust. Inspired by tools like siege and wrk, warmer provides advanced features including sitemap-based cache warming, JavaScript site crawling, and asset discovery (css, js, images etc) with concurrent request handling.

Quick Start

# Pull and run warmer with Docker (works on x86_64 and ARM64)
docker run abhaisasidharan/warmer warmer https://example.com -t5S -c10

# For JavaScript/WASM sites
docker run abhaisasidharan/warmer warmer https://example-spa.com -j

# For sitemap-based cache warming
docker run abhaisasidharan/warmer warmer https://example.com -s -t1M -c25

For Developers

# Build and push latest version (both architectures)
./build-docker.sh

# Build specific version
./build-docker.sh v1.0.0

# Build locally for testing
./build-docker.sh v1.0.0 --local

Features

• Parallel URL Processing: Each thread processes different URLs from the sitemap in parallel with dynamic thread pool scaling
• Time-based Testing: Run tests for specific durations (seconds, minutes, hours)
• Advanced Sitemap Support: Finds sitemap URLs from robots.txt and handles sitemap indexes
• Single URL Testing: Test individual URLs like siege
• Asset Loading: Automatically loads CSS, JS, and images from HTML pages
• Internet Mode: Random URL selection for realistic load testing
• Crawl Mode: Process each URL only once, perfect for cache warming
• Follow Links Mode: Automatically discover and test URLs by following links from the provided URL
• Siege-like Output: Colored status codes, actual HTTP version, and comprehensive statistics
• Performance Metrics: Transaction rate, throughput, response times, availability
• Cloudflare Bypass: Rotating user agents and realistic request patterns to avoid bot detection

Usage

Command Line Options

• -c, --concurrent <NUM>: Number of concurrent users (default: 25)
• -t, --time <TIME>: Time to run the test (e.g., 5S, 1M, 1H)
• -r, --repetitions <NUM>: Number of repetitions per user
• -d, --delay <SECONDS>: Delay between requests (default: 0)
• -v, --verbose: Verbose output
• -s, --sitemap: Use sitemap mode (default for all modes)
• -i, --internet: Internet mode - random URL selection from sitemap
• -n, --no-assets: Disable static asset loading (CSS, JS, images) from HTML pages
• -w, --crawl: Crawl mode - process each URL only once, then stop (uses concurrency 1, automatically uses sitemap)
• -f, --follow-links: Follow links mode - discover URLs by following links from the provided URL (bypasses sitemap processing)
• -j, --js: JavaScript mode - use headless Chrome browser to crawl JavaScript/WASM sites and discover dynamically generated links (automatically disables sitemap mode)
• -T, --discovery-threads <NUM>: Number of discovery threads for JavaScript mode (default: CPU cores / 2, min 2, max 8)
• -C, --config <PATH>: Path to TOML config file (e.g. warmer-config.toml). All options except URL can be set in the config. CLI always overrides config for every option except user-agent settings (user-agent, user-agents), which are long and are taken from the config file only.
• -a, --anonymize: Rotate through the built-in browser-like User-Agent list (anonymize requests). To use a custom list, set user-agent (single) or user-agents (list) in the config file.

Examples

Single URL load testing:

docker run abhaisasidharan/warmer warmer https://example.com -t5S -c10

Sitemap-based cache warming:

docker run abhaisasidharan/warmer warmer https://example.com -s -t1M -c25

Internet mode with random URL selection:

docker run abhaisasidharan/warmer warmer https://example.com -s -i -t30S -c50

Crawl mode (cache warming - each URL once):

# Crawl all URLs from sitemap once (automatically detects sitemap)
docker run abhaisasidharan/warmer warmer https://example.com -w

# Or explicitly specify sitemap URL
docker run abhaisasidharan/warmer warmer https://example.com/sitemap.xml -w

Pure load testing without assets:

docker run abhaisasidharan/warmer warmer https://abh.ai -t5S -n

Verbose mode with asset loading:

docker run abhaisasidharan/warmer warmer https://abh.ai -t30S -c10 -v

Follow links mode (for sites without sitemap.xml):

docker run abhaisasidharan/warmer warmer https://www.tdtreedays.com -f

JavaScript mode (for JS/WASM sites):

docker run abhaisasidharan/warmer warmer https://example-spa.com -j

JavaScript mode with custom thread count:

docker run abhaisasidharan/warmer warmer https://example-spa.com -j -T4

Sitemap mode (XML parsing only):

docker run abhaisasidharan/warmer warmer https://example.com -s

Using a config file:

# Copy the example config and edit as needed
cp warmer-config.toml.example warmer-config.toml
warmer https://example.com -C warmer-config.toml

Installation

Docker (Recommended)

The easiest way to run warmer is using Docker:

# Pull the latest image
docker pull abhaisasidharan/warmer

# Run warmer with any URL
docker run abhaisasidharan/warmer warmer https://abh.ai -t5S -c10

# Run with JavaScript mode
docker run abhaisasidharan/warmer warmer https://example.com -j -T4

# Run with sitemap mode
docker run abhaisasidharan/warmer warmer https://example.com -s -t1M -c25

Install from Package (.deb or .rpm)

Debian/Ubuntu (.deb)

1. Download the .deb package from the releases page
2. Install the package:

   sudo dpkg -i warmer_*.deb

3. If there are missing dependencies, install them:

   sudo apt-get install -f

Fedora/RHEL (.rpm)

1. Download the .rpm package from the releases page
2. Install the package:

   sudo dnf install warmer-*.rpm

Google Chrome for JavaScript Mode

The --js flag requires Google Chrome to be installed. Install it as follows:

For Debian/Ubuntu:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb

For Fedora/RHEL:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
sudo dnf install ./google-chrome-stable_current_x86_64.rpm

Note: Without Chrome installed, the --js flag will not work. Regular sitemap and crawling modes do not require Chrome.

Build from source

If you prefer to build from source:

1. Clone the repo: git clone https://github.com/codingsasi/warmer.git && cd warmer
2. Install Rust: https://doc.rust-lang.org/cargo/getting-started/installation.html
3. Build: cargo build --release
4. The binary will be in target/release/warmer
5. You may need to install: libudev-dev, libssl-dev, openssl, pkg-config, build-essential

Output Example

Checking robots.txt at https://abh.ai/robots.txt
Found 1 sitemap URL(s) in robots.txt
Processing sitemap: https://abh.ai/sitemap.xml
Found 102 URLs in sitemap
Total unique URLs found across all sitemaps: 99
** WARMER 0.1.2
** Preparing 25 concurrent users for battle.
The server is now under load...
HTTP/2.0 200     0.10 secs: 4 KB ==> GET  /
HTTP/2.0 200     0.28 secs: 191 KB ==> GET  /pagead/js/adsbygoogle.js
HTTP/2.0 200     0.30 secs: 191 KB ==> GET  /pagead/js/adsbygoogle.js
HTTP/2.0 200     0.29 secs: 0 bytes ==> GET  /fe_bg.wasm
HTTP/2.0 200     0.30 secs: 191 KB ==> GET  /pagead/js/adsbygoogle.js
HTTP/2.0 200     0.31 secs: 0 bytes ==> GET  /beacon.min.js
Load testing completed...

Transactions:                  1 hits
Availability:             100.00 %
Elapsed time:               0.24 secs
Data transferred:           0.00 MB
Response time:            103.00 ms
Transaction rate:           4.17 trans/sec
Throughput:                 0.01 MB/sec
Concurrency:                0.43
Successful transactions:        1
Failed transactions:           0
Longest transaction:      103.00 ms
Shortest transaction:     103.00 ms

Notes

• Docker Recommended: The easiest way to run warmer is using Docker. No local installation needed!
• Package Installation: Native .deb and .rpm packages are available for direct installation on Linux systems
• Multi-architecture Support: Works on x86_64 (Intel/AMD) and ARM64 (Apple Silicon, ARM servers)
• Chrome Required for JS Mode: The --js flag requires Google Chrome to be installed separately (see Installation section)
• Large sitemaps that include other zipped or gzipped sitemaps are not supported yet
• Asset loading is enabled by default for comprehensive cache warming
• Use -n, --no-assets for pure load testing without asset crawling
• The tool automatically checks robots.txt to find the correct sitemap URL
• Sitemap indexes (XML files containing links to other sitemaps) are fully supported and recursively processed
• JavaScript mode uses multiple headless Chrome instances for parallel discovery - use -T, --discovery-threads to control memory usage
• Cross-platform: Works on Linux, macOS, and Windows with Docker