scfuzzbench

Appearance

Run gh-29739216061-1 ​

Trial run

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Charts ​

Bugs Over TimeTime To KFinal DistributionPlateau And Late ShareInvariant Overlap (UpSet)Coverage Over TimeCorpus Size Over TimeSeq/s Over TimeTx/s Over TimeGas/s Over TimeCPU Usage Over TimeMemory Usage Over Time

Report ​

Fuzzer Benchmark Report (from bug-count CSV) ​

  • Time budget: 4.00h

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Executive summary ​

This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

Bugs found at fixed time budgets (median [IQR]) ​

FuzzerRuns1h4h
recon-fuzzer28 [7,8]9 [9,9]
echidna26 [6,7]7 [7,7]
foundry26 [6,7]6 [6,7]
medusa26 [6,6]6 [6,7]

Overall metrics ​

FuzzerAUC (norm)Plateau timeLate discovery shareFinal medianFinal IQR
recon-fuzzer0.8961.50h0.00090.00
echidna0.7191.20h0.00070.00
foundry0.6780.90h0.00060.50
medusa0.6472.80h0.07760.50

Milestones: time-to-k and success rates ​

Fuzzertime-to-1 (p50)time-to-3 (p50)time-to-5 (p50)reach-1 ratereach-3 ratereach-5 rate
recon-fuzzer0.00h0.10h0.40h100.0%100.0%100.0%
echidna0.10h0.10h0.30h100.0%100.0%100.0%
foundry0.10h0.10h0.15h100.0%100.0%100.0%
medusa0.10h0.10h0.25h100.0%100.0%100.0%

Throughput metrics (if supported by log format) ​

Values are run-level rates aggregated per fuzzer; n/a indicates the parser could not recover that metric from logs.

FuzzerRunsTx/s runsTx/s p50 [p25,p75]Gas/s runsGas/s p50 [p25,p75]
recon-fuzzer224049.21 [3960.51,4137.91]217521558.50 [15392029.25,19651087.75]
echidna222021.31 [1979.67,2062.95]28108372492.50 [4054186246.25,12162558738.75]
foundry2211671.27 [11650.25,11692.30]27989702792.82 [7696331835.63,8283073750.01]
medusa221270.00 [1261.50,1278.50]2149613096.00 [147741559.50,151484632.50]

Progress metrics from logs (fuzzer-specific proxies) ​

Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.

FuzzerRunsSeq/s runsSeq/s p50 [p25,p75]Corpus runsCorpus p50 [p25,p75]
recon-fuzzer20n/a283.00 [80.50,85.50]
echidna20n/a2263.50 [232.75,294.25]
foundry20n/a20.00 [0.00,0.00]
medusa2211.50 [11.25,11.75]2228.50 [228.25,228.75]

Coverage over time (opt-in, fuzzer-native signals) ​

  • Status: enabled
  • Source-based coverage is preferred, but the current timestamped progress formats expose only native bytecode/instrumentation proxies. The signals below have different units and must not be ranked, pooled, or plotted on a shared scale.
  • Foundry showmap artifacts are a separate, opt-in post-campaign edge replay; they are not real-time source coverage.
FuzzerNative signalSource-based?Live availabilityRuns with signalTemporal observationsObserved parser provenanceFinal native signal p50 [p25,p75]Limitation
recon-fuzzerUnique instructions (instructions)No (runtime proxy)End-of-run Recon summary (not a live series)2/2end-of-run only; 2 sample(s); excluded from time-series charttext-metrics:unique_instructions71810.50 [71581.75,72039.25]Tool/version-specific final instruction count; no source locations or intermediate observations.
echidnacov coverage points (coverage points)No (runtime proxy)Echidna status lines2/2live; 9519 samples; first 0.03-0.03h, last 4.00-4.00h of 4.00h budgettext-metrics:cov119496.00 [109098.00,129894.00]Tool/config-specific counter; no source locations in the progress log.
foundrycumulative_edges_seen (instrumented edges)No (runtime proxy)Compatible Foundry JSON pulse builds2/2live; 5726 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budgetjson-metrics:cumulative_edges_seen0.00 [0.00,0.00]Edge identities depend on the build/instrumentation; showmap replay is separate and post-campaign.
medusabranches hit (branches)No (runtime proxy)Medusa status lines2/2live; 9498 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budgettext-metrics:branches_hit5041.00 [5016.50,5065.50]Branch identities depend on compiled bytecode/config; no source locations.

Function selector sanity checks ​

Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in selector_distribution.csv; selector_summary.json preserves statuses and provenance.

  • Expected-set status: available (23 selector(s))
  • Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth
FuzzerStatusInstances with telemetrySaved-corpus callsUnique selectors
echidnaavailable2/25193044
foundryunavailable0/200
medusaavailable2/22348263
recon-fuzzeravailable2/21643632
Most common saved-corpus selectors ​
FuzzerSelectorFunctionCallsShare
echidna0xae22f4b8iAaveOracle_setPrice(uint256,uint256)20283.91%
echidna0xf24a44c9assert_canary_ASSERTION_CANARY(uint256)18213.51%
echidna0x5c9d5cb7iSpoke_supply_ASSERTION_SUPPLY_DOS(uint256,uint256)17513.37%
echidna0xe6fb1960iSpoke_updateReserveConfig(uint256,(uint24,bool,bool,bool,bool,bool))17383.35%
echidna0x4b43b035invariant_totalBorrowedLessThanSupplied_v0()16983.27%
echidna0xd3fabfaciSpoke_withdraw_ASSERTION_WITHDRAW_DOS(uint256,uint256)16903.25%
echidna0xf9f2ec54iSpoke_borrow(uint256,uint256)16863.25%
echidna0xf7d31edcinvariant_hubAddedAssetsGreaterThanSpokeSum()16743.22%
echidna0x7aca1cb2invariant_supplySharePriceAndDrawnIndexMonotonic()16703.22%
echidna0xb722c6ediHub_updateAssetConfig(uint256,(address,uint16,address,address),(uint16,uint32,uint32,uint32))16603.20%
medusa0x5c9d5cb7iSpoke_supply_ASSERTION_SUPPLY_DOS(uint256,uint256)11995.11%
medusa0xa89ef612iSpoke_updateUserDynamicConfig()11895.06%
medusa0xb722c6ediHub_updateAssetConfig(uint256,(address,uint16,address,address),(uint16,uint32,uint32,uint32))11374.84%
medusa0xdfbe4d80iSpoke_repay_ASSERTION_REPAY_DOS(uint256,uint256)11254.79%
medusa0x5d55852eiSpoke_addDynamicReserveConfig(uint256,(uint16,uint32,uint16))10994.68%
medusa0x6801b82eswitchActor(uint256)10924.65%
medusa0xae22f4b8iAaveOracle_setPrice(uint256,uint256)10844.62%
medusa0x20bea9ddiHub_mintFeeShares_ASSERTION_MINT_FEE_SHARES_PPS_CHANGE(uint256)10734.57%
medusa0x8aa3bc37switch_oracle(uint256)10294.38%
medusa0x256aeedeiSpoke_updateLiquidationConfig((uint128,uint64,uint16))10184.34%
recon-fuzzer0x27dd847ciSpoke_liquidationCall_ASSERTION_LIQUIDATION_CALL_DOS(uint256,uint256,uint256,uint256,bool)6994.25%
recon-fuzzer0x4896f9c3iHub_updateSpokeConfig(uint256,uint256,(uint40,uint40,uint24,bool,bool))6634.03%
recon-fuzzer0x7aca1cb2invariant_supplySharePriceAndDrawnIndexMonotonic()6483.94%
recon-fuzzer0x256aeedeiSpoke_updateLiquidationConfig((uint128,uint64,uint16))6303.83%
recon-fuzzer0xca2664f9iSpoke_setUsingAsCollateral(uint256,bool)6053.68%
recon-fuzzer0x5d55852eiSpoke_addDynamicReserveConfig(uint256,(uint16,uint32,uint16))5973.63%
recon-fuzzer0x66d9a9a0targetArtifactSelectors()5773.51%
recon-fuzzer0x44a0611biHub_setInterestRateData(uint256,(uint16,uint32,uint32,uint32))5743.49%
recon-fuzzer0x77494a8einvariant_hubAddedSharesMatchesSpokeSum()5733.49%
recon-fuzzer0x6801b82eswitchActor(uint256)5703.47%
Selector diagnostic findings ​
  • INFO: i-05e6717728e1919df-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0xb0464fdc
  • INFO: i-0e67ab59a18b228f6-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0xb0464fdc
Selector telemetry limitations ​
  • i-0a9505372fea882f4-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution
  • i-0cc6af8d1beaaef2e-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution

Statistical comparison (Mann-Whitney U and Vargha-Delaney A12) ​

Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts. The tests are two-sided, with Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

Warnings:

  • One or more fuzzers have fewer than 5 runs. Statistical power may be limited.
Fuzzer AFuzzer BMedian AMedian BU statisticA12 (A over B)Effect magnitudep-valuep (corrected)Significant
recon-fuzzerechidna974.01.000large0.19391.0000no
recon-fuzzerfoundry96.54.01.000large0.22071.0000no
recon-fuzzermedusa96.54.01.000large0.22071.0000no
echidnafoundry76.53.00.750large0.61711.0000no
echidnamedusa76.53.00.750large0.61711.0000no
foundrymedusa6.56.52.00.500negligible1.00001.0000no

No pairwise comparison reached significance after Bonferroni correction.

Note: A12 is the probability that a randomly selected Fuzzer A run has a higher final bug count than a randomly selected Fuzzer B run, counting ties as half. Values above 0.5 favor A, values below 0.5 favor B, and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5 are negligible (<0.06), small (<0.14), medium (<0.21), and large (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power. A12 and its magnitude label are descriptive; the thresholds are rules of thumb. Neither the effect size nor statistical significance establishes practical importance, causation, or performance beyond the observed runs.

Shape-based interpretation (rules of thumb) ​

  • Fast-start / early-plateau: high early checkpoint median + early plateau time + low late discovery share.
  • Steady: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
  • Slow-burn / late-surge: low early checkpoints but high late discovery share and later plateau time; often higher final median.

Limitations ​

  • Core count-based charts use normalized event identities stored under the legacy bugs_found column. They do not count crash inputs, but they are not confirmed root-cause bug counts.
  • Use known_bug_report.md, known_bug_summary.csv, and known_bug_findings.csv for evidence-backed known-bug hit rates; use broken_invariants.md / broken_invariants.csv for raw invariant identities.
  • Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
  • Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.

Broken invariants ​

  • Budget filter: 4.00h
  • Events considered: 58 / 58
  • Unique invariants: 11

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Per-fuzzer totals ​

FuzzerInvariants
echidna8
foundry7
medusa7
recon-fuzzer10

High-level overlap ​

  • Shared by all active fuzzers: 6
  • Exclusive to echidna: 0
  • Exclusive to foundry: 1
  • Exclusive to medusa: 0
  • Exclusive to recon-fuzzer: 2

Grouped invariants ​

Exclusive to echidna (0)

None.

Exclusive to foundry (1)
  • invariant_totalBorrowedLessThanSupplied_v2
Exclusive to medusa (0)

None.

Exclusive to recon-fuzzer (2)
  • iSpoke_liquidationCall
  • iSpoke_repay
Shared by all active fuzzers (6)
  • assert_canary
  • iHub_mintFeeShares
  • invariant_canary
  • invariant_shouldNotBecomeLiquidatable
  • invariant_totalBorrowedLessThanSupplied_v0
  • invariant_totalBorrowedLessThanSupplied_v1

Top shared subsets (top 2 by size):

echidna, medusa, recon-fuzzer (1)
  • iSpoke_withdraw
echidna, recon-fuzzer (1)
  • invariant_supplySharePriceAndDrawnIndexMonotonic

Ground-truth known-bug mapping ​

  • Target: aave-v4
  • Evidence-pinned commit: 459b020058f4a65d18eb1481083b75c766340124
  • Resolution: matched run repository and commit
  • Catalog: benchmarks/known_bugs.json

Only the two instrumented health-check canaries are mapped. Other property failures remain unmapped until a reviewed root-cause record ties them to a confirmed bug.

Canaries are reported separately as harness-health checks and are never included in the known-bug hit-rate denominator.

Per-fuzzer results ​

FuzzerReplicatesKnown-bug replicate hitsCatalog coverageCanary replicate hitsUnmapped event findings
echidna2n/an/a4/4 (100.0%)10
foundry2n/an/a4/4 (100.0%)9
medusa2n/an/a4/4 (100.0%)9
recon-fuzzer2n/an/a4/4 (100.0%)14

Known-bug catalog ​

Ground-truth IDKindTitleEvidence
None cataloged

Unmapped event identities ​

FuzzerEvent identityReplicates observed
echidnaiHub_mintFeeShares2
echidnaiSpoke_withdraw1
echidnainvariant_shouldNotBecomeLiquidatable1
echidnainvariant_supplySharePriceAndDrawnIndexMonotonic2
echidnainvariant_totalBorrowedLessThanSupplied_v02
echidnainvariant_totalBorrowedLessThanSupplied_v12
foundryiHub_mintFeeShares2
foundryinvariant_shouldNotBecomeLiquidatable2
foundryinvariant_totalBorrowedLessThanSupplied_v02
foundryinvariant_totalBorrowedLessThanSupplied_v12
foundryinvariant_totalBorrowedLessThanSupplied_v21
medusaiHub_mintFeeShares2
medusaiSpoke_withdraw1
medusainvariant_shouldNotBecomeLiquidatable2
medusainvariant_totalBorrowedLessThanSupplied_v02
medusainvariant_totalBorrowedLessThanSupplied_v12
recon-fuzzeriHub_mintFeeShares2
recon-fuzzeriSpoke_liquidationCall2
recon-fuzzeriSpoke_repay2
recon-fuzzeriSpoke_withdraw1
recon-fuzzerinvariant_shouldNotBecomeLiquidatable1
recon-fuzzerinvariant_supplySharePriceAndDrawnIndexMonotonic2
recon-fuzzerinvariant_totalBorrowedLessThanSupplied_v02
recon-fuzzerinvariant_totalBorrowedLessThanSupplied_v12

Counting semantics ​

  • A canonical known-bug ID counts at most once per replicate, even when several event aliases or counterexamples reach it.
  • A replicate is identified by (run_id, instance_id, fuzzer).
  • Known-bug hit rate is canonical bug/replicate hits ÷ (cataloged bugs × replicates).
  • Unmapped rows are distinct normalized event identities per replicate. They are triage candidates, not claimed bugs.
  • Crash inputs and corpus files are not used as bug identities.

Runner resource usage ​

  • Budget filter: 4.00h
  • Instances with metrics: 8
  • Total samples: 22409

Per-fuzzer medians (across instances) ​

FuzzerInstancesCPU active avg (%)CPU active peak (%)Memory used avg (GiB)Memory used peak (GiB)Memory used avg (%)Memory used peak (%)
echidna295.5498.1519.0619.4762.2163.52
foundry299.16100.001.053.723.4112.14
medusa298.2499.782.053.746.7012.19
recon-fuzzer296.97100.002.444.577.9514.93

Instance stats ​

InstanceFuzzerSamplesDuration (h)CPU active avg (%)CPU active peak (%)Memory avg (GiB)Memory peak (GiB)Memory avg (%)Memory peak (%)
i-07bcdb0ec5e6923ae-echidna-v2.3.1echidna27844.0095.4198.1319.0719.4862.2263.56
i-0f43de17950e8eb72-echidna-v2.3.1echidna27804.0095.6798.1619.0619.4562.1963.47
i-0a9505372fea882f4-foundry-git-02c05d9foundry28094.0099.16100.001.053.723.4112.13
i-0cc6af8d1beaaef2e-foundry-git-02c05d9foundry28094.0099.16100.001.053.723.4212.15
i-073d0d97b602f0292-medusa-v1.4.1medusa28094.0098.2999.832.313.727.5512.13
i-0a98815a185f5f4b5-medusa-v1.4.1medusa28084.0098.1999.721.793.765.8512.26
i-05e6717728e1919df-recon-v0.4.6recon-fuzzer28054.0096.75100.002.454.327.9814.10
i-0e67ab59a18b228f6-recon-v0.4.6recon-fuzzer28054.0097.19100.002.434.837.9215.76

Manifest ​

  • scfuzzbench_commit: c92d80466d7a9669770064cf9b922e95b2413892
  • target_repo_url: https://github.com/scfuzzbench/aave-v4-scfuzzbench
  • target_commit: 459b020058f4a65d18eb1481083b75c766340124
  • benchmark_type: property
  • instance_type: c6a.4xlarge
  • instances_per_fuzzer: 2
  • timeout_hours: 4
  • aws_region: us-east-1
  • ubuntu_ami_id: ami-052355af2a014bd2c
  • foundry_version: 1.7.2-dev
  • foundry_git_repo: https://github.com/foundry-rs/foundry
  • foundry_git_ref: 02c05d970d2801da0aef8b82486ce84b01ede36d
  • foundry_source_patch: scfuzzbench-throughput-progress-v1@sha256:2ee9e69b77c8007c78c816eb9ca791684aa5ecede0651b63f86cdd2e055eb17e
  • echidna_version: 2.3.1
  • medusa_version: 1.4.1
  • recon_version: 0.4.6
  • fuzzer_keys: echidna, foundry, medusa, recon-fuzzer

Artifacts ​

Fully static. Generated in CI from S3 run artifacts.

Copyright © Recon-Fuzz.