Run gh-30404285801-1
- Date (UTC):
2026-07-28 22:22:27Z - Benchmark:
1242c7e572d9e09a8bbd29bcf98a0fa8 - Timeout:
0.5h - Target:
scfuzzbench/aave-v4-scfuzzbench@6f97958083 - Benchmark type:
property - Instances:
1×c6a.8xlargeper fuzzer - Harness:
scfuzzbench/scfuzzbench@af0281945b
Trial run
Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.
Charts
Report
Fuzzer Benchmark Report (from bug-count CSV)
- Time budget: 0.50h
Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.
Executive summary
This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.
Bugs found at fixed time budgets (median [IQR])
| Fuzzer | Runs | 0.5h |
|---|---|---|
| echidna | 1 | 8 [8,8] |
| medusa | 1 | 8 [8,8] |
Overall metrics
| Fuzzer | AUC (norm) | Plateau time | Late discovery share | Final median | Final IQR |
|---|---|---|---|---|---|
| echidna | 0.700 | 0.30h | 0.250 | 8 | 0.00 |
| medusa | 0.650 | 0.50h | 0.125 | 8 | 0.00 |
Milestones: time-to-k and success rates
| Fuzzer | time-to-1 (p50) | time-to-3 (p50) | time-to-5 (p50) | reach-1 rate | reach-3 rate | reach-5 rate |
|---|---|---|---|---|---|---|
| echidna | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| medusa | 0.00h | 0.10h | 0.20h | 100.0% | 100.0% | 100.0% |
Throughput metrics (if supported by log format)
Values are run-level rates aggregated per fuzzer; n/a indicates the parser could not recover that metric from logs.
| Fuzzer | Runs | Tx/s runs | Tx/s p50 [p25,p75] | Gas/s runs | Gas/s p50 [p25,p75] |
|---|---|---|---|---|---|
| echidna | 1 | 1 | 3975.95 [3975.95,3975.95] | 1 | 34731700385.00 [34731700385.00,34731700385.00] |
| medusa | 1 | 1 | 2247.00 [2247.00,2247.00] | 1 | 276800320.00 [276800320.00,276800320.00] |
Progress metrics from logs (fuzzer-specific proxies)
Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.
| Fuzzer | Runs | Seq/s runs | Seq/s p50 [p25,p75] | Corpus runs | Corpus p50 [p25,p75] |
|---|---|---|---|---|---|
| echidna | 1 | 0 | n/a | 1 | 94.00 [94.00,94.00] |
| medusa | 1 | 1 | 21.00 [21.00,21.00] | 1 | 226.00 [226.00,226.00] |
Function selector sanity checks
Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in selector_distribution.csv; selector_summary.json preserves statuses and provenance.
- Expected-set status: available (23 selector(s))
- Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth
| Fuzzer | Status | Instances with telemetry | Saved-corpus calls | Unique selectors |
|---|---|---|---|---|
| echidna | available | 1/1 | 9242 | 44 |
| medusa | available | 1/1 | 11134 | 66 |
Most common saved-corpus selectors
| Fuzzer | Selector | Function | Calls | Share |
|---|---|---|---|---|
| echidna | 0x27dd847c | iSpoke_liquidationCall_ASSERTION_LIQUIDATION_CALL_DOS(uint256,uint256,uint256,uint256,bool) | 408 | 4.41% |
| echidna | 0x50fb9ec5 | invariant_totalBorrowedSharesMatchesSpokeSum() | 371 | 4.01% |
| echidna | 0x4b430dd2 | iSpoke_updateDynamicReserveConfig(uint256,uint24,(uint16,uint32,uint16)) | 364 | 3.94% |
| echidna | 0x4896f9c3 | iHub_updateSpokeConfig(uint256,uint256,(uint40,uint40,uint24,bool,bool)) | 358 | 3.87% |
| echidna | 0x44a0611b | iHub_setInterestRateData(uint256,(uint16,uint32,uint32,uint32)) | 350 | 3.79% |
| echidna | 0x0a9254e4 | setUp() | 340 | 3.68% |
| echidna | 0xe6fb1960 | iSpoke_updateReserveConfig(uint256,(uint24,bool,bool,bool,bool,bool)) | 332 | 3.59% |
| echidna | 0xf24a44c9 | assert_canary_ASSERTION_CANARY(uint256) | 332 | 3.59% |
| echidna | 0xb722c6ed | iHub_updateAssetConfig(uint256,(address,uint16,address,address),(uint16,uint32,uint32,uint32)) | 330 | 3.57% |
| echidna | 0x5c9d5cb7 | iSpoke_supply_ASSERTION_SUPPLY_DOS(uint256,uint256) | 328 | 3.55% |
| medusa | 0xb722c6ed | iHub_updateAssetConfig(uint256,(address,uint16,address,address),(uint16,uint32,uint32,uint32)) | 569 | 5.11% |
| medusa | 0x5c9d5cb7 | iSpoke_supply_ASSERTION_SUPPLY_DOS(uint256,uint256) | 555 | 4.98% |
| medusa | 0xf9f2ec54 | iSpoke_borrow(uint256,uint256) | 527 | 4.73% |
| medusa | 0x4b430dd2 | iSpoke_updateDynamicReserveConfig(uint256,uint24,(uint16,uint32,uint16)) | 525 | 4.72% |
| medusa | 0x55ba98ff | switch_asset(uint256) | 514 | 4.62% |
| medusa | 0x20bea9dd | iHub_mintFeeShares_ASSERTION_MINT_FEE_SHARES_PPS_CHANGE(uint256) | 505 | 4.54% |
| medusa | 0x4896f9c3 | iHub_updateSpokeConfig(uint256,uint256,(uint40,uint40,uint24,bool,bool)) | 500 | 4.49% |
| medusa | 0x6801b82e | switchActor(uint256) | 498 | 4.47% |
| medusa | 0xa89ef612 | iSpoke_updateUserDynamicConfig() | 497 | 4.46% |
| medusa | 0xca2664f9 | iSpoke_setUsingAsCollateral(uint256,bool) | 496 | 4.45% |
Statistical comparison (Mann-Whitney U and Vargha-Delaney A12)
Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts. The tests are two-sided, with Bonferroni correction applied for 0 comparison(s). Significance level: alpha = 0.05.
Warnings:
- One or more fuzzers have fewer than 5 runs. Statistical power may be limited.
- Skipping echidna vs medusa: fewer than 2 runs.
Note: A12 is the probability that a randomly selected Fuzzer A run has a higher final bug count than a randomly selected Fuzzer B run, counting ties as half. Values above 0.5 favor A, values below 0.5 favor B, and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5 are negligible (<0.06), small (<0.14), medium (<0.21), and large (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power. A12 and its magnitude label are descriptive; the thresholds are rules of thumb. Neither the effect size nor statistical significance establishes practical importance, causation, or performance beyond the observed runs.
Shape-based interpretation (rules of thumb)
- Fast-start / early-plateau: high early checkpoint median + early plateau time + low late discovery share.
- Steady: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
- Slow-burn / late-surge: low early checkpoints but high late discovery share and later plateau time; often higher final median.
Limitations
- Core count-based charts use normalized event identities stored under the legacy
bugs_foundcolumn. They do not count crash inputs, but they are not confirmed root-cause bug counts. - Use
known_bug_report.md,known_bug_summary.csv, andknown_bug_findings.csvfor evidence-backed known-bug hit rates; usebroken_invariants.md/broken_invariants.csvfor raw invariant identities. - Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
- Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.
Broken invariants
- Budget filter: 0.50h
- Events considered: 16 / 16
- Unique invariants: 9
Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.
Per-fuzzer totals
| Fuzzer | Invariants |
|---|---|
| echidna | 8 |
| medusa | 8 |
High-level overlap
- Shared by all active fuzzers: 7
- Exclusive to
echidna: 1 - Exclusive to
medusa: 1
Grouped invariants
Exclusive to echidna (1)
iSpoke_liquidationCall
Exclusive to medusa (1)
invariant_totalBorrowedLessThanSupplied_v1
Shared by all active fuzzers (7)
assert_canaryiHub_mintFeeSharesiSpoke_repayiSpoke_withdrawinvariant_canaryinvariant_shouldNotBecomeLiquidatableinvariant_totalBorrowedLessThanSupplied_v0
Ground-truth known-bug mapping
Ground-truth mapping was not applied.
- Reason: run commit 6f9795808367de1357c1059377dde9201b9c1d66 is not the evidence-pinned catalog revision 459b020058f4a65d18eb1481083b75c766340124
- Catalog:
benchmarks/known_bugs.json
No event was classified as a known bug. Raw event analysis remains available, but it must not be interpreted as a confirmed bug count.
Runner resource usage
- Budget filter: 0.50h
- Instances with metrics: 2
- Total samples: 701
Per-fuzzer medians (across instances)
| Fuzzer | Instances | CPU active avg (%) | CPU active peak (%) | Memory used avg (GiB) | Memory used peak (GiB) | Memory used avg (%) | Memory used peak (%) |
|---|---|---|---|---|---|---|---|
| echidna | 1 | 82.61 | 94.83 | 33.09 | 37.17 | 53.80 | 60.43 |
| medusa | 1 | 86.81 | 98.36 | 2.36 | 4.22 | 3.83 | 6.86 |
Instance stats
| Instance | Fuzzer | Samples | Duration (h) | CPU active avg (%) | CPU active peak (%) | Memory avg (GiB) | Memory peak (GiB) | Memory avg (%) | Memory peak (%) |
|---|---|---|---|---|---|---|---|---|---|
| i-08651eea8b0663884-echidna-v2.3.3 | echidna | 349 | 0.50 | 82.61 | 94.83 | 33.09 | 37.17 | 53.80 | 60.43 |
| i-00206d91e8f69c004-medusa-v1.5.1 | medusa | 352 | 0.50 | 86.81 | 98.36 | 2.36 | 4.22 | 3.83 | 6.86 |
Manifest
- scfuzzbench_commit:
af0281945bfd4945f2fa1e2d6594319ba6707048 - target_repo_url: https://github.com/scfuzzbench/aave-v4-scfuzzbench
- target_commit:
6f9795808367de1357c1059377dde9201b9c1d66 - benchmark_type:
property - instance_type:
c6a.8xlarge - instances_per_fuzzer:
1 - timeout_hours:
0.5 - aws_region:
us-east-1 - ubuntu_ami_id:
ami-052355af2a014bd2c - foundry_version:
1.7.2-dev - foundry_git_repo:
https://github.com/foundry-rs/foundry - foundry_git_ref:
02c05d970d2801da0aef8b82486ce84b01ede36d - foundry_source_patch:
scfuzzbench-throughput-progress-v1@sha256:2ee9e69b77c8007c78c816eb9ca791684aa5ecede0651b63f86cdd2e055eb17e - echidna_version:
2.3.3 - medusa_version:
1.5.1 - recon_version:
0.4.18-fix - fuzzer_keys:
echidna, medusa
Artifacts
Manifest (index): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/runs/gh-30404285801-1/1242c7e572d9e09a8bbd29bcf98a0fa8/manifest.json
Report prefix: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/
Broken invariants (Markdown): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/broken_invariants.md
Broken invariants (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/broken_invariants.csv
Ground-truth known bugs (Markdown): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/known_bug_report.md
Ground-truth summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/known_bug_summary.csv
Ground-truth findings (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/known_bug_findings.csv
Throughput summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/throughput_summary.csv
Progress metrics summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/progress_metrics_summary.csv
Function selector distribution (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/selector_distribution.csv
Function selector summary and health (JSON): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/selector_summary.json
Runner resource usage (Markdown): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/runner_resource_usage.md
Runner resource summary (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/runner_resource_summary.csv
Runner resource timeseries (CSV): https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/runner_resource_timeseries.csv
Analysis bundle: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/analysis/1242c7e572d9e09a8bbd29bcf98a0fa8/gh-30404285801-1/bundles/analysis.zip
Raw logs prefix: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/logs/gh-30404285801-1/1242c7e572d9e09a8bbd29bcf98a0fa8/
Raw corpus prefix: https://scfuzzbench-logs-185f44d6.s3.us-east-1.amazonaws.com/corpus/gh-30404285801-1/1242c7e572d9e09a8bbd29bcf98a0fa8/