Run 1783632734
- Date (UTC):
2026-07-09 21:32:14Z - Benchmark:
e24bf1814e919efc02c7f23e3f501e1e - Timeout:
2h - Target:
scfuzzbench/origin-dollar-scfuzzbench@299ec6b6bf - Benchmark type:
property - Instances:
4×c6a.4xlargeper fuzzer - Harness:
scfuzzbench/scfuzzbench@263edb652d
Trial run
Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.
Charts
Report
Fuzzer Benchmark Report (from bug-count CSV)
- Time budget: 2.00h
Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.
Executive summary
This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.
Bugs found at fixed time budgets (median [IQR])
| Fuzzer | Runs | 1h |
|---|---|---|
| echidna | 4 | 14 [14,15] |
| medusa | 4 | 15 [15,15] |
| recon-fuzzer | 4 | 15 [15,15] |
| foundry | 4 | 14 [13,14] |
Overall metrics
| Fuzzer | AUC (norm) | Plateau time | Late discovery share | Final median | Final IQR |
|---|---|---|---|---|---|
| echidna | 0.878 | 1.10h | 0.065 | 16 | 1.25 |
| medusa | 0.894 | 0.10h | 0.000 | 15 | 0.00 |
| recon-fuzzer | 0.877 | 0.80h | 0.000 | 15 | 0.25 |
| foundry | 0.844 | 0.00h | 0.000 | 14 | 1.25 |
Milestones: time-to-k and success rates
| Fuzzer | time-to-1 (p50) | time-to-3 (p50) | time-to-5 (p50) | reach-1 rate | reach-3 rate | reach-5 rate |
|---|---|---|---|---|---|---|
| echidna | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| medusa | 0.00h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| recon-fuzzer | 0.00h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| foundry | 0.00h | 0.00h | 0.00h | 100.0% | 100.0% | 100.0% |
Throughput metrics (if supported by log format)
Values are run-level rates aggregated per fuzzer; n/a indicates the parser could not recover that metric from logs.
| Fuzzer | Runs | Tx/s runs | Tx/s p50 [p25,p75] | Gas/s runs | Gas/s p50 [p25,p75] |
|---|---|---|---|---|---|
| echidna | 4 | 4 | 10874.37 [10851.64,10965.26] | 4 | 86036224433.50 [63097240917.00,88043342966.50] |
| medusa | 4 | 4 | 48267.00 [47936.75,48591.50] | 4 | 2559509458.50 [2551632794.50,2564507917.25] |
| recon-fuzzer | 4 | 4 | 3219.90 [3160.13,3277.67] | 4 | 4998565.00 [4747070.75,5602409.50] |
Progress metrics from logs (fuzzer-specific proxies)
Coverage/corpus values are parsed from each fuzzer's native progress output and are useful for trend context, not strict cross-fuzzer equivalence.
| Fuzzer | Runs | Seq/s runs | Seq/s p50 [p25,p75] | Coverage runs | Coverage p50 [p25,p75] | Corpus runs | Corpus p50 [p25,p75] |
|---|---|---|---|---|---|---|---|
| echidna | 4 | 0 | n/a | 4 | 15380.00 [15373.50,15385.00] | 4 | 30.00 [30.00,30.50] |
| medusa | 4 | 4 | 482.00 [478.75,485.50] | 4 | 1546.00 [1546.00,1546.00] | 4 | 152.50 [151.50,153.75] |
| recon-fuzzer | 4 | 0 | n/a | 4 | 16466.00 [16465.25,16466.00] | 4 | 29.00 [28.00,30.00] |
Statistical comparison (Mann-Whitney U test)
Pairwise Mann-Whitney U tests on end-of-budget bug counts (two-sided). Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.
Warnings:
- One or more fuzzers have fewer than 5 runs. Statistical power may be limited.
| Fuzzer A | Fuzzer B | Median A | Median B | U statistic | p-value | p (corrected) | Significant |
|---|---|---|---|---|---|---|---|
| echidna | medusa | 15.5 | 15 | 10.0 | 0.6171 | 1.0000 | no |
| echidna | recon-fuzzer | 15.5 | 15 | 11.0 | 0.4357 | 1.0000 | no |
| echidna | foundry | 15.5 | 13.5 | 14.0 | 0.1038 | 0.6225 | no |
| medusa | recon-fuzzer | 15 | 15 | 10.0 | 0.4533 | 1.0000 | no |
| medusa | foundry | 15 | 13.5 | 14.0 | 0.0668 | 0.4005 | no |
| recon-fuzzer | foundry | 15 | 13.5 | 13.0 | 0.1606 | 0.9635 | no |
No pairwise comparison reached significance after Bonferroni correction.
Note: The Mann-Whitney U test assesses whether one fuzzer tends to find more bugs than another across runs. It does not measure effect size or practical importance. Small sample sizes (fewer than 5 runs) reduce statistical power.
Shape-based interpretation (rules of thumb)
- Fast-start / early-plateau: high early checkpoint median + early plateau time + low late discovery share.
- Steady: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
- Slow-burn / late-surge: low early checkpoints but high late discovery share and later plateau time; often higher final median.
Limitations
- Core metrics in this section are count-based; use
broken_invariants.md/broken_invariants.csvfor invariant identities. - Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
- Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.
Broken invariants
- Budget filter: 2.00h
- Events considered: 235 / 235
- Unique invariants: 16
Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.
Per-fuzzer totals
| Fuzzer | Invariants |
|---|---|
| echidna | 16 |
| foundry | 15 |
| medusa | 15 |
| recon-fuzzer | 15 |
High-level overlap
- Shared by all active fuzzers: 15
- Exclusive to
echidna: 1 - Exclusive to
foundry: 0 - Exclusive to
medusa: 0 - Exclusive to
recon-fuzzer: 0
Grouped invariants
Exclusive to echidna (1)
AssertionFailed
Exclusive to foundry (0)
None.
Exclusive to medusa (0)
None.
Exclusive to recon-fuzzer (0)
None.
Shared by all active fuzzers (15)
assert_canaryinvariant_canarytestBurnBalancetestChangeSupplytestMintBalanceRoundingtestOptInBalancetestOptInBalanceRoundingtestRebasingCreditsPerTokenNotIncreasedtestTotalSupplyLessThanTotalBalancetestTransferBalanceReceivedLessRoundingtestTransferBalanceReceivedMoretestTransferBalanceSentLesstestTransferFromShouldNotReverttestTransferSelftestTransferWithinBalanceDoesNotRevert
Runner resource usage
- Budget filter: 2.00h
- Instances with metrics: 16
- Total samples: 22731
Per-fuzzer medians (across instances)
| Fuzzer | Instances | CPU active avg (%) | CPU active peak (%) | Memory used avg (GiB) | Memory used peak (GiB) | Memory used avg (%) | Memory used peak (%) |
|---|---|---|---|---|---|---|---|
| echidna | 4 | 85.03 | 97.34 | 24.69 | 29.07 | 80.56 | 94.87 |
| foundry | 4 | 6.29 | 18.18 | 0.94 | 1.22 | 3.06 | 3.97 |
| medusa | 4 | 86.63 | 88.23 | 1.05 | 1.36 | 3.42 | 4.43 |
| recon-fuzzer | 4 | 99.54 | 100.00 | 1.51 | 1.75 | 4.92 | 5.72 |
Instance stats
| Instance | Fuzzer | Samples | Duration (h) | CPU active avg (%) | CPU active peak (%) | Memory avg (GiB) | Memory peak (GiB) | Memory avg (%) | Memory peak (%) |
|---|---|---|---|---|---|---|---|---|---|
| i-01b07a70af8a3b68d-echidna-v2.3.1 | echidna | 1429 | 2.00 | 85.19 | 97.18 | 23.57 | 27.92 | 76.92 | 91.12 |
| i-023604732418ea0ff-echidna-v2.3.1 | echidna | 1429 | 2.00 | 85.57 | 97.31 | 25.04 | 29.38 | 81.71 | 95.87 |
| i-080101dc9cc279059-echidna-v2.3.1 | echidna | 1323 | 1.85 | 84.86 | 97.37 | 24.40 | 28.82 | 79.63 | 94.04 |
| i-0d7dc81c9cbb80978-echidna-v2.3.1 | echidna | 1366 | 1.91 | 84.86 | 97.60 | 24.97 | 29.33 | 81.49 | 95.69 |
| i-01e753f71c390af72-foundry-git-907ba08 | foundry | 1436 | 2.00 | 6.28 | 18.18 | 0.94 | 1.19 | 3.07 | 3.87 |
| i-047d6cb522a28e5d4-foundry-git-907ba08 | foundry | 1436 | 2.00 | 6.28 | 18.18 | 0.94 | 1.10 | 3.05 | 3.59 |
| i-0a1b90a61f9b07d0a-foundry-git-907ba08 | foundry | 1436 | 2.00 | 6.29 | 18.18 | 0.94 | 1.25 | 3.06 | 4.08 |
| i-0d5b8ea4806915256-foundry-git-907ba08 | foundry | 1436 | 2.00 | 6.29 | 25.00 | 0.94 | 1.26 | 3.05 | 4.12 |
| i-05730f751290dc059-medusa-v1.4.1 | medusa | 1433 | 2.00 | 86.93 | 88.29 | 1.09 | 1.40 | 3.54 | 4.57 |
| i-0781befd8cc6c62ab-medusa-v1.4.1 | medusa | 1433 | 2.00 | 86.50 | 88.18 | 1.03 | 1.34 | 3.37 | 4.36 |
| i-081a2a08d04c26d12-medusa-v1.4.1 | medusa | 1433 | 2.00 | 86.76 | 90.16 | 1.06 | 1.38 | 3.44 | 4.51 |
| i-093f5af352e00fb17-medusa-v1.4.1 | medusa | 1433 | 2.00 | 86.45 | 88.05 | 1.04 | 1.25 | 3.39 | 4.09 |
| i-0212fa473c32bbca2-recon-v0.4.6 | recon-fuzzer | 1427 | 2.00 | 99.52 | 100.00 | 1.50 | 1.84 | 4.89 | 6.01 |
| i-048ecbfa72534759b-recon-v0.4.6 | recon-fuzzer | 1427 | 2.00 | 99.55 | 100.00 | 1.51 | 1.70 | 4.92 | 5.55 |
| i-0811af5ea4ca950c6-recon-v0.4.6 | recon-fuzzer | 1427 | 2.00 | 99.57 | 100.00 | 1.52 | 1.81 | 4.95 | 5.90 |
| i-0a5470097bfa1653b-recon-v0.4.6 | recon-fuzzer | 1427 | 2.00 | 99.50 | 100.00 | 1.50 | 1.66 | 4.91 | 5.42 |
Manifest
- scfuzzbench_commit:
263edb652de84d394c9548a2780dba676fc1da00 - target_repo_url: https://github.com/scfuzzbench/origin-dollar-scfuzzbench
- target_commit:
299ec6b6bf5401b43efc3e9f7ee5cab3e76167c4 - benchmark_type:
property - instance_type:
c6a.4xlarge - instances_per_fuzzer:
4 - timeout_hours:
2 - aws_region:
us-east-1 - ubuntu_ami_id:
ami-0a02a779008fa3b99 - foundry_version:
v1.7.1 - foundry_git_repo:
https://github.com/foundry-rs/foundry - foundry_git_ref:
907ba0817e8feb2404da9d506a12b69834cfb393 - echidna_version:
2.3.1 - medusa_version:
1.4.1 - recon_version:
0.4.6 - fuzzer_keys:
echidna, foundry, medusa, recon-fuzzer
Artifacts
Manifest (index): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/runs/1783632734/e24bf1814e919efc02c7f23e3f501e1e/manifest.json
Report prefix: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/
Broken invariants (Markdown): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/broken_invariants.md
Broken invariants (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/broken_invariants.csv
Throughput summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/throughput_summary.csv
Progress metrics summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/progress_metrics_summary.csv
Runner resource usage (Markdown): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/runner_resource_usage.md
Runner resource summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/runner_resource_summary.csv
Runner resource timeseries (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/runner_resource_timeseries.csv
Analysis bundle: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/bundles/analysis.zip
Corpus bundle: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/e24bf1814e919efc02c7f23e3f501e1e/1783632734/bundles/corpus.zip
Raw logs prefix: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/logs/1783632734/e24bf1814e919efc02c7f23e3f501e1e/
Raw corpus prefix: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/corpus/1783632734/e24bf1814e919efc02c7f23e3f501e1e/