English | 中文文档
yuantest-playwright is a comprehensive Playwright test orchestration, execution, and reporting platform with real-time Web Dashboard visualization. It provides intelligent test management capabilities that help teams efficiently manage, analyze, and debug E2E tests at scale.
Zero Learning Curve · Zero Migration Cost · Pure Playwright Ecosystem
@slow, @flaky, @skip annotations and custom tags| Advantage | Benefit |
|---|---|
| All-in-One Solution | Complete test lifecycle management without combining multiple tools |
| No Internal API Dependencies | Executes via Playwright CLI, ensuring upgrade compatibility |
| Real-time Monitoring | WebSocket push for live test progress visibility |
| Intelligent Flaky Management | Automatic detection, quarantine, and statistical analysis |
| AI Failure Analysis | Smart categorization with targeted fix suggestions |
| Internationalization | Built-in Chinese/English support |
yuantest-playwright provides complete test lifecycle management without combining multiple tools:
| Module | Description |
|---|---|
| Orchestrator | Smart test sharding, load balancing, optimized allocation based on historical execution time |
| Executor | Execution via Playwright CLI, no internal API dependencies, strong upgrade compatibility |
| RealtimeReporter | WebSocket real-time push of test progress |
| DashboardServer | Complete Web UI + REST API |
| FlakyTestManager | Automatic detection, quarantine, and statistics of unstable tests |
| ArtifactManager | Unified management of screenshots, videos, and trace files |
This is the core differentiating value of the project, a rare capability in the market:
// Automatically record test history and calculate failure rate
existing.failureRate = failures / totalRuns;
// One-click quarantine support
yuantest flaky --quarantine <test-id>
// ShardOptimizer - Optimize sharding based on historical execution time
const optimizedAssignments = await optimizer.optimize(tests, shards);
// Execute via Playwright CLI, not internal APIs
spawn('npx', ['playwright', 'test', ...args]);
This means:
| Dimension | yuantest-playwright | allure-playwright |
|---|---|---|
| Learning Curve | ✅ Zero learning curve - parameters identical to Playwright CLI | ⚠️ Need to learn Allure configuration and annotations |
| Migration Cost | ✅ Zero migration cost - pure Playwright commands | ⚠️ Need to configure Allure Server and History |
| Ecosystem Dependency | ✅ Pure Playwright ecosystem - GPL-3.0 license | ⚠️ Depends on Allure ecosystem |
| Positioning | Full-stack test management platform | Report generator |
| Real-time | ✅ WebSocket real-time push | ❌ Generate report after test completion |
| Web Dashboard | ✅ Built-in React Dashboard | ✅ Allure Server (requires extra deployment) |
| Flaky Management | ✅ Auto detection + quarantine + statistics | ❌ None |
| Test Orchestration | ✅ Smart sharding + load balancing | ❌ None |
| Test Execution | ✅ Built-in Executor | ❌ Requires external execution |
| Native Playwright Integration | ✅ Executes via Playwright CLI, no internal API | ✅ Integrates via Reporter API |
| AI Failure Analysis | ✅ Auto categorization + fix suggestions | ❌ None |
| Aggregated Reports | ✅ Built-in, works out of the box | ⚠️ Requires Allure Server configuration |
| Historical Trends | ✅ Built-in storage, multi-dimensional trends | ✅ Requires History configuration |
| Failure Analysis | ✅ Auto categorization + fix suggestions | ⚠️ Manual analysis required |
| Internationalization | ✅ Chinese/English | ⚠️ Self-configuration required |
| Deployment Complexity | ✅ Single npm package | ⚠️ Requires Allure Server |
| Report Aesthetics | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| CI/CD Integration | ✅ CLI + API | ✅ Wide support |
| yuantest-playwright Advantages | allure-playwright Advantages |
|---|---|
| All-in-one solution | More visually appealing reports |
| Real-time test progress monitoring | Mature ecosystem, large community |
| Flaky test management | Support for multiple test frameworks |
| Intelligent test orchestration | Rich historical trend charts |
| Simple deployment | Strong plugin extensibility |
Use yuantest-playwright for daily test management and real-time monitoring, while integrating allure-playwright for generating polished final reports:
# Daily development: Use yuantest for real-time monitoring
yuantest run --test-dir ./
# CI/CD: Generate Allure report for archiving
npx playwright test --reporter=allure-playwright
# Global installation
npm install -g yuantest-playwright
# Or as project dependency
npm install --save-dev yuantest-playwright
git clone https://github.com/yuandiv/yuantest-playwright.git
cd yuantest-playwright
npm install
npm run build
npm link
Get started in 30 seconds:
# Install
npm install -g yuantest-playwright
# Run tests
yuantest run --test-dir ./
# Start Web Dashboard
yuantest ui
# Analyze flaky tests
yuantest flaky
# AI-powered failure diagnosis
yuantest analyze --id <run-id> --ai
# Basic usage
yuantest run --test-dir ./
# Specify project name and output directory
yuantest run --project my-app --test-dir ./e2e --output ./reports
# Parallel execution with 4 shards
yuantest run --test-dir ./ --shards 4
# Specify multiple browsers
yuantest run --test-dir ./ --browsers chromium,firefox
# Set timeout and retries
yuantest run --test-dir ./ --timeout 60000 --retries 2
# Default port 5274
yuantest ui
# Custom port
yuantest ui --port 8080
# Custom report and data directories
yuantest ui --port 5274 --output ./reports --data ./test-data
Then open http://localhost:5274 in your browser to view the visualization interface.
yuantest --help
yuantest run --help
yuantest ui --help
| Parameter | Short | Description | Default |
|---|---|---|---|
--project |
-p |
Project name | test-project |
--test-dir |
-t |
Test file directory | ./ |
--output |
-o |
Output directory | ./test-output |
--shards |
-s |
Number of shards | 1 |
--workers |
-w |
Number of workers | 1 |
--browsers |
-b |
Browser list (comma-separated) | chromium |
--base-url |
Base URL | ||
--timeout |
Timeout (ms) | 30000 | |
--retries |
Number of retries | 0 | |
--grep |
Run matching tests | ||
--update-snapshots |
Update snapshots | false | |
--config |
-c |
Config file path | |
--trace |
Trace mode: off, on, retain-on-failure, on-first-retry | on-first-retry | |
--screenshot |
Screenshot mode: off, on, only-on-failure | only-on-failure | |
--video |
Video mode: off, on, retain-on-failure, on-first-retry | retain-on-failure | |
--tags |
Run only tests with these tags (comma-separated) | ||
--project-filter |
Run only specific browser project | ||
--visual-threshold |
Visual diff threshold (0-1) | 0.2 | |
--annotations |
Enable annotation scanning | false | |
--html-report |
Generate Playwright HTML report | true |
# View test shard distribution plan
yuantest orchestrate --test-dir ./ --shards 4
# View recent 10 reports
yuantest report --limit 10
# View specific report
yuantest report --id run_20240101_120000_abc123
# View Flaky statistics
yuantest flaky
# List all Flaky tests
yuantest flaky --list
# List quarantined tests
yuantest flaky --quarantined
# Quarantine a specific test
yuantest flaky --quarantine <test-id>
# Release a specific test
yuantest flaky --release <test-id>
# Custom threshold
yuantest flaky --list --threshold 0.5
# Analyze failure reasons for a specific run
yuantest analyze --id run_20240101_120000_abc123
# Enable AI diagnosis
yuantest analyze --id run_20240101_120000_abc123 --ai
# Cluster analysis on failures
yuantest analyze --id run_20240101_120000_abc123 --cluster
# Filter by category
yuantest analyze --id run_20240101_120000_abc123 --filter timeout
# List all traces
yuantest trace --list
# Open a trace file in the viewer
yuantest trace --view <trace-file>
# Show trace statistics
yuantest trace --stats
# Clean traces older than 7 days
yuantest trace --clean
# Scan test annotations
yuantest annotations --test-dir ./
# Scan test tags
yuantest tags --test-dir ./
# List all artifacts
yuantest artifacts --list
# Show artifact statistics
yuantest artifacts --stats
# Clean artifacts older than 7 days
yuantest artifacts --clean
# Show visual testing statistics
yuantest visual --stats
# Update all baselines with current screenshots
yuantest visual --update
# Open Playwright HTML report in browser
yuantest show-report
# Specify report path
yuantest show-report --path ./reports/html-report
# View test run history
yuantest test-history <testId>
# Output in JSON format
yuantest test-history <testId> --json
# List all error patterns
yuantest error-patterns --list
# List custom error patterns only
yuantest error-patterns --custom
# Add a custom error pattern
yuantest error-patterns --add '<json>'
# Show current LLM configuration
yuantest llm-config --show
# Test LLM connection
yuantest llm-config --test
# Check LLM status
yuantest llm-config --status
# Update LLM configuration
yuantest llm-config --set '<json>'
# View test health metrics
yuantest health
# Output in JSON format
yuantest health --json
# List high-risk tests
yuantest prediction --high-risk
# View prediction for a specific test
yuantest prediction --test <testId>
# View duration anomalies
yuantest prediction --duration-anomalies
# View test correlation analysis
yuantest correlations
# Show causal graph summary
yuantest correlations --causal-graph
# Rerun a specific test from a previous run
yuantest rerun <runId> <testId>
After starting, visit http://localhost:<port>, which includes the following modules:
Dashboard provides RESTful API (prefix: /api/v1/) for easy integration with other systems. For the complete API reference, see Documentation.
| Method | Path | Description |
|---|---|---|
| GET | /api/v1/health |
Health check |
| GET | /api/v1/stats |
Overall statistics |
| GET | /api/v1/runs |
Run list (paginated) |
| GET | /api/v1/runs/:id |
Run details |
| POST | /api/v1/runs |
Start a test run |
| POST | /api/v1/runs/stop |
Stop current run |
| DELETE | /api/v1/runs/:id |
Delete a run |
| Method | Path | Description |
|---|---|---|
| GET | /api/v1/flaky |
Flaky test list |
| GET | /api/v1/flaky/quarantined |
Quarantined tests |
| POST | /api/v1/flaky/:testId/quarantine |
Quarantine test |
| POST | /api/v1/flaky/:testId/release |
Release test |
| GET | /api/v1/flaky/stats |
Flaky statistics |
| GET | /api/v1/flaky/health |
Overall health score |
| GET | /api/v1/flaky/predictions/high-risk |
High-risk predictions |
| Method | Path | Description |
|---|---|---|
| GET | /api/v1/analysis/:runId |
Failure analysis |
| GET | /api/v1/failures/analysis |
Cross-run failure summary |
| POST | /api/v1/diagnosis |
AI diagnosis |
| POST | /api/v1/diagnosis/stream |
AI diagnosis (SSE stream) |
| POST | /api/v1/diagnosis/cluster |
Cluster diagnosis |
| Method | Path | Description |
|---|---|---|
| GET | /api/v1/tests |
Discovered tests |
| POST | /api/v1/runs/:runId/tests/:testId/rerun |
Rerun a test |
| POST | /api/v1/runs/:runId/batch-rerun |
Batch rerun tests |
| GET | /api/v1/tests/:testId/history |
Test history |
| Method | Path | Description |
|---|---|---|
| GET | /api/v1/traces |
Trace list |
| GET | /api/v1/artifacts |
Artifact list |
| GET | /api/v1/attachments/file |
Serve attachment file |
import {
Orchestrator,
Executor,
Reporter,
FlakyTestManager,
DashboardServer,
} from 'yuantest-playwright';
async function main() {
// 1. Orchestrate tests
const orchestrator = new Orchestrator({
version: 'my-app',
testDir: './e2e',
outputDir: './reports',
shards: 4,
browsers: ['chromium', 'firefox'],
});
await orchestrator.initialize();
const plan = await orchestrator.orchestrate();
// 2. Execute tests
const executor = new Executor(orchestrator.getConfig());
// Listen to events
executor.on('run_started', (data) => {
console.log(`Run started: ${data.runId}`);
});
executor.on('test_result', (result) => {
console.log(`[${result.status}] ${result.title} (${result.duration}ms)`);
});
executor.on('run_progress', (progress) => {
console.log(`Progress: ${progress.passed}/${progress.totalTests} passed`);
});
executor.on('output', (data) => {
process.stdout.write(data.data);
});
executor.on('run_completed', async (result) => {
// 3. Generate report
const reporter = new Reporter('./reports');
const reportPath = await reporter.generateReport(result);
console.log(`Report: ${reportPath}`);
// 4. Analyze failures
const analysis = await reporter.analyzeFailures(result);
console.log(`Failures: ${analysis.length}`);
});
const result = await executor.execute({
grepPattern: 'smoke',
projectFilter: 'chromium',
updateSnapshots: false,
});
console.log(`Final: ${result.passed}/${result.totalTests} passed`);
// 5. Start Dashboard
const server = new DashboardServer(5274, './reports', './test-data');
await server.start();
}
main();
import {
FlakyTestManager,
AnnotationManager,
TagManager,
TraceManager,
ArtifactManager,
VisualTestingManager,
} from 'yuantest-playwright';
// Flaky test management
const flakyManager = new FlakyTestManager('./test-data');
// Get flaky tests
const flakyTests = flakyManager.getFlakyTests(0.3);
console.log(`Found ${flakyTests.length} flaky tests`);
// Quarantine test
await flakyManager.quarantineTest('test-id-123');
// Annotation management
const annotationManager = new AnnotationManager();
const annotations = await annotationManager.scanDirectory('./');
console.log(`Found ${annotations.length} annotated tests`);
// Tag management
const tagManager = new TagManager();
const tags = await tagManager.scanDirectory('./');
// Trace management
const traceManager = new TraceManager(
{ enabled: true, mode: 'on', screenshots: true, snapshots: true, sources: true, attachments: true },
'./traces'
);
// Artifact management
const artifactManager = new ArtifactManager(
{ enabled: true, screenshots: 'on', videos: 'on' },
'./artifacts'
);
// Visual testing
const visualManager = new VisualTestingManager(
{ enabled: true, threshold: 0.2, maxDiffPixelRatio: 0.01, maxDiffPixels: 10, updateSnapshots: false },
'./visual-testing'
);
| Module | Description |
|---|---|
Orchestrator, ShardOptimizer |
Test orchestration and smart sharding |
Executor, ParallelExecutor |
Test execution |
Reporter, JSONReporter |
Report generation |
RealtimeReporter, RealtimeReporterClient |
WebSocket real-time reporting |
FlakyTestManager |
Flaky test detection, quarantine, and statistics |
DashboardServer |
Web UI + REST API server |
TraceManager |
Playwright trace management |
AnnotationManager |
Test annotation scanning |
TagManager |
Test tag management |
ArtifactManager |
Screenshot, video, and trace artifact management |
VisualTestingManager |
Pixel-comparison visual regression testing |
PlaywrightConfigBuilder |
Playwright config generation |
loadConfigFile, mergeConfig |
Configuration loading and merging |
StorageProvider, FilesystemStorage |
Storage abstraction |
TestDiscovery |
Test file discovery |
LRUCache, TTLCache |
Caching utilities |
yuantest-playwright/
├── bin/
│ ├── cli.js # CLI entry
│ └── start-ui.js # Dashboard startup script
├── dashboard/ # Web Dashboard frontend source
│ ├── src/
│ │ ├── components/ # React components
│ │ ├── hooks/ # Custom Hooks
│ │ ├── services/ # API services
│ │ └── types/ # TypeScript types
│ └── index.html
├── src/
│ ├── index.ts # Main entry
│ ├── types/ # Type definitions
│ ├── orchestrator/ # Test orchestrator
│ ├── executor/ # Test executor
│ ├── reporter/ # Report generator
│ ├── realtime/ # Real-time reporting
│ ├── flaky/ # Flaky management
│ ├── diagnosis/ # AI diagnosis
│ ├── config/ # Configuration management
│ ├── trace/ # Trace management
│ ├── annotations/ # Annotation scanner
│ ├── tags/ # Tag management
│ ├── artifacts/ # Artifact management
│ ├── visual/ # Visual testing
│ ├── storage/ # Storage providers
│ ├── cache/ # LRU/TTL cache
│ ├── base/ # Base manager classes
│ ├── discovery/ # Test discovery
│ ├── middleware/ # Express middleware
│ ├── validation/ # Zod validation
│ ├── utils/ # Utility functions
│ ├── i18n/ # Internationalization
│ ├── constants/ # Constants
│ ├── logger/ # Logger module
│ ├── cli/ # CLI commands
│ └── ui/ # Dashboard server
├── documentation/ # Documentation source
├── tests/ # Test files
└── package.json
# GitHub Actions example
name: E2E Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install dependencies
run: npm ci
- name: Run E2E tests
run: |
npm install -g yuantest-playwright
yuantest run --test-dir ./e2e --shards 4 --output ./reports
- name: Upload reports
uses: actions/upload-artifact@v3
with:
name: test-reports
path: reports/
# Accelerate large test suites with smart sharding
yuantest run --test-dir ./e2e --shards 8 --workers 4
# Quarantine flaky tests to avoid blocking CI
yuantest flaky --quarantine-all
yuantest run --test-dir ./e2e
# Run tests on all browsers
yuantest run --test-dir ./e2e --browsers chromium,firefox,webkit
# Run on specific browser only
yuantest run --test-dir ./e2e --browsers chromium
Support for customizing behavior through configuration files:
// yuantest.config.ts
import type { TestConfig } from 'yuantest-playwright';
const config: TestConfig = {
version: 'my-app',
testDir: './e2e',
outputDir: './reports',
shards: 4,
browsers: ['chromium', 'firefox'],
timeout: 60000,
retries: 2,
traces: {
enabled: true,
mode: 'on-first-retry',
screenshots: true,
snapshots: true,
sources: true,
attachments: true,
},
artifacts: {
enabled: true,
screenshots: 'only-on-failure',
videos: 'retain-on-failure',
},
visualTesting: {
enabled: true,
threshold: 0.2,
maxDiffPixelRatio: 0.01,
maxDiffPixels: 10,
updateSnapshots: false,
},
};
export default config;
Contributions are welcome! Feel free to submit code, report issues, or make suggestions!
git checkout -b feature/amazing-feature)git commit -m 'Add amazing feature')git push origin feature/amazing-feature)See CONTRIBUTING.md for details.
GPL-3.0 © YuanDiv
Thanks to the following open source projects:
No. All parameters of yuantest-playwright are identical to Playwright CLI, so your existing Playwright knowledge can be reused directly. A Web UI is also provided for use without command line.
No. yuantest-playwright uses pure Playwright commands to execute tests with no proprietary APIs. You can switch back to native Playwright anytime without modifying any test code - truly zero migration cost.
Fully open source. GPL-3.0 license, pure Playwright ecosystem, no barriers. Built on Playwright native capabilities, no compatibility issues with version upgrades.
Fully compatible. Tests are executed via Playwright CLI, supporting all native features: Trace, screenshots, videos, snapshots, etc. Automatically collects and manages all Playwright artifacts.
AI-powered analysis. Automatically categorizes failure reasons (timeout, assertion failure, element not found, etc.), provides fix suggestions, supports failure trend analysis, and helps quickly identify root causes.
If this project helps you, please give it a ⭐️ Star!
Made with ❤️ by YuanDiv