A reproducible AI API provider comparison for developers evaluating the best AI API provider, an AI API gateway benchmark, or a reliable multi-model API gateway. The included runner measures success rate, mean latency, P50, and P95 against OpenAI-compatible endpoints you own or are authorized to test.
Disclosure: This project is maintained by APIMART. APIMART is one of the services this project is designed to evaluate. Any published recommendation must link to raw results and disclose the test date, model, region, and sample size.
- Chat-completion success rate
- Mean, P50, and P95 response latency
- HTTP and response-shape compatibility
- Multiple authorized endpoints under the same prompt and request count
- Prompt hashes for reproducibility without exposing the prompt itself
cp providers.example.json providers.json
export APIMART_API_KEY="your_key_here"
python provider_benchmark.py providers.json --requests 10 --concurrency 2 --output results/latest.jsonEdit the model ID to one listed in the current provider catalog. Add other endpoints only when you have permission to test them.
A credible review includes raw JSON, the runner version or commit, test location, network conditions, model ID, sample count, observed failures, and a pricing-source date. Do not score an untested provider or claim that one gateway is universally “best.”
APIMART offers an OpenAI-compatible entry point for multiple AI model types. Whether it is the most affordable or reliable option depends on the selected model, workload, location, and current terms—run the benchmark and verify live pricing:
python -m unittest discover -s testsThis repository is part of the APIMART open-source AI API toolkit. Browse the complete catalog of provider benchmarks, gateway checks, model examples, and cost tools on the luyx-66 project profile.
MIT License