scfuzzbench

Appearance

Run gh-29759131085-1 ​

Trial run

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Charts ​

Bugs Over TimeTime To KFinal DistributionPlateau And Late ShareInvariant Overlap (UpSet)Coverage Over TimeCorpus Size Over TimeSeq/s Over TimeTx/s Over TimeGas/s Over TimeCPU Usage Over TimeMemory Usage Over Time

Report ​

Fuzzer Benchmark Report (from bug-count CSV) ​

  • Time budget: 4.00h

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Executive summary ​

This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

Bugs found at fixed time budgets (median [IQR]) ​

FuzzerRuns1h4h
medusa215 [15,15]15 [15,15]
echidna214 [14,15]15 [14,16]
recon-fuzzer214 [14,14]14 [14,14]
foundry214 [13,14]14 [14,14]

Overall metrics ​

FuzzerAUC (norm)Plateau timeLate discovery shareFinal medianFinal IQR
medusa0.9160.10h0.000150.00
echidna0.9021.40h0.000151.00
recon-fuzzer0.8560.20h0.000140.00
foundry0.8441.20h0.000140.00

Milestones: time-to-k and success rates ​

Fuzzertime-to-1 (p50)time-to-3 (p50)time-to-5 (p50)reach-1 ratereach-3 ratereach-5 rate
medusa0.00h0.10h0.10h100.0%100.0%100.0%
echidna0.10h0.10h0.10h100.0%100.0%100.0%
recon-fuzzer0.00h0.05h0.10h100.0%100.0%100.0%
foundry0.10h0.10h0.10h100.0%100.0%100.0%

Throughput metrics (if supported by log format) ​

Values are run-level rates aggregated per fuzzer; n/a indicates the parser could not recover that metric from logs.

FuzzerRunsTx/s runsTx/s p50 [p25,p75]Gas/s runsGas/s p50 [p25,p75]
medusa2247991.00 [47830.50,48151.50]22514082134.00 [2513608802.00,2514555466.00]
echidna225596.10 [5543.79,5648.41]227230235923.50 [13615117961.75,40845353885.25]
recon-fuzzer22857.44 [853.21,861.66]24945544.50 [4156940.75,5734148.25]
foundry22185197.04 [182087.17,188306.90]213993386024.76 [13951408468.82,14035363580.70]

Progress metrics from logs (fuzzer-specific proxies) ​

Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.

FuzzerRunsSeq/s runsSeq/s p50 [p25,p75]Corpus runsCorpus p50 [p25,p75]
medusa22479.50 [478.25,480.75]2206.50 [204.25,208.75]
echidna20n/a226.50 [26.25,26.75]
recon-fuzzer20n/a222.00 [21.50,22.50]
foundry20n/a20.00 [0.00,0.00]

Coverage over time (opt-in, fuzzer-native signals) ​

  • Status: enabled
  • Source-based coverage is preferred, but the current timestamped progress formats expose only native bytecode/instrumentation proxies. The signals below have different units and must not be ranked, pooled, or plotted on a shared scale.
  • Foundry showmap artifacts are a separate, opt-in post-campaign edge replay; they are not real-time source coverage.
FuzzerNative signalSource-based?Live availabilityRuns with signalTemporal observationsObserved parser provenanceFinal native signal p50 [p25,p75]Limitation
medusabranches hit (branches)No (runtime proxy)Medusa status lines2/2live; 9592 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budgettext-metrics:branches_hit1546.00 [1546.00,1546.00]Branch identities depend on compiled bytecode/config; no source locations.
echidnacov coverage points (coverage points)No (runtime proxy)Echidna status lines2/2live; 9574 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budgettext-metrics:cov15377.00 [15373.00,15381.00]Tool/config-specific counter; no source locations in the progress log.
recon-fuzzerUnique instructions (instructions)No (runtime proxy)End-of-run Recon summary (not a live series)2/2end-of-run only; 2 sample(s); excluded from time-series charttext-metrics:unique_instructions16459.50 [16457.75,16461.25]Tool/version-specific final instruction count; no source locations or intermediate observations.
foundrycumulative_edges_seen (instrumented edges)No (runtime proxy)Compatible Foundry JSON pulse builds2/2live; 5734 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budgetjson-metrics:cumulative_edges_seen0.00 [0.00,0.00]Edge identities depend on the build/instrumentation; showmap replay is separate and post-campaign.

Function selector sanity checks ​

Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in selector_distribution.csv; selector_summary.json preserves statuses and provenance.

  • Expected-set status: available (46 selector(s))
  • Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth
FuzzerStatusInstances with telemetrySaved-corpus callsUnique selectors
echidnaavailable2/2528947
foundryunavailable0/200
medusaavailable2/21160746
recon-fuzzeravailable2/2421845
Most common saved-corpus selectors ​
FuzzerSelectorFunctionCallsShare
echidna0xd5afe6cdoptIn(uint8)1683.18%
echidna0xd89cecc4testNonRebasingSupplyVsTotalSupply()1572.97%
echidna0xf24a44c9assert_canary_ASSERTION_CANARY(uint256)1452.74%
echidna0xdad9f8f0approve(uint8,uint8,uint256)1402.65%
echidna0x0a1fc5e7testMintBalance(uint8,uint256)1392.63%
echidna0x43a45e75testTransferBalanceSentLess(uint8,uint8,uint256)1382.61%
echidna0x6900d46etransfer(uint8,uint8,uint256)1372.59%
echidna0xc7df1adatestTransferFromShouldNotRevert(uint8,uint8,uint8,uint256)1352.55%
echidna0x1a042ae7invariant_canary()1322.50%
echidna0x178e5c32testChangeSupply(uint256)1302.46%
medusa0x00a3c5d1testTransferWithinBalanceDoesNotRevert(uint8,uint8,uint256)4153.58%
medusa0x9164cd26testTransferBalanceSentMore(uint8,uint8,uint256)4053.49%
medusa0x657808a3testBalanceOfShouldNotRevert(uint8)4043.48%
medusa0x379db993testTransferBalanceReceivedLessRounding(uint8,uint8,uint256)3643.14%
medusa0xc7df1adatestTransferFromShouldNotRevert(uint8,uint8,uint8,uint256)3583.08%
medusa0x324b65f5testApprove(uint8,uint8,uint256)3503.02%
medusa0x43a45e75testTransferBalanceSentLess(uint8,uint8,uint256)3242.79%
medusa0x9de07e5atestOptInBalanceRounding(uint8)3222.77%
medusa0x7fc05f0atestOptOutTotalSupply(uint8)3092.66%
medusa0x6dcc9dcctestRebasingCreditsPerTokenNotIncreased()3012.59%
recon-fuzzer0x13ffd369testBurnBalance(uint8,uint256)1694.01%
recon-fuzzer0x0a1fc5e7testMintBalance(uint8,uint256)1373.25%
recon-fuzzer0xe4840d4ftestTransferBalanceReceivedMore(uint8,uint8,uint256)1283.03%
recon-fuzzer0x43a45e75testTransferBalanceSentLess(uint8,uint8,uint256)1273.01%
recon-fuzzer0x178e5c32testChangeSupply(uint256)1232.92%
recon-fuzzer0x72a96677transferFrom(uint8,uint8,uint8,uint256)1232.92%
recon-fuzzer0x8f287234testBurnAllBalanceShouldNotRevert(uint8)1082.56%
recon-fuzzer0xd89cecc4testNonRebasingSupplyVsTotalSupply()1072.54%
recon-fuzzer0x379db993testTransferBalanceReceivedLessRounding(uint8,uint8,uint256)1032.44%
recon-fuzzer0x7fc05f0atestOptOutTotalSupply(uint8)1032.44%
Selector diagnostic findings ​
  • INFO: i-00286b19f02049aa2-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 2 selector(s): 0x12b58349, 0x54113357
  • INFO: i-0922c2f8649781881-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 2 selector(s): 0x12b58349, 0x54113357
Selector telemetry limitations ​
  • i-024a9959ae72deb9f-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution
  • i-0eb2f56ba7c7e57aa-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution

Statistical comparison (Mann-Whitney U and Vargha-Delaney A12) ​

Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts. The tests are two-sided, with Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

Warnings:

  • One or more fuzzers have fewer than 5 runs. Statistical power may be limited.
Fuzzer AFuzzer BMedian AMedian BU statisticA12 (A over B)Effect magnitudep-valuep (corrected)Significant
medusaechidna15152.00.500negligible1.00001.0000no
medusarecon-fuzzer15144.01.000large0.19391.0000no
medusafoundry15144.01.000large0.19391.0000no
echidnarecon-fuzzer15143.00.750large0.61711.0000no
echidnafoundry15143.00.750large0.61711.0000no
recon-fuzzerfoundry14142.00.500negligible1.00001.0000no

No pairwise comparison reached significance after Bonferroni correction.

Note: A12 is the probability that a randomly selected Fuzzer A run has a higher final bug count than a randomly selected Fuzzer B run, counting ties as half. Values above 0.5 favor A, values below 0.5 favor B, and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5 are negligible (<0.06), small (<0.14), medium (<0.21), and large (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power. A12 and its magnitude label are descriptive; the thresholds are rules of thumb. Neither the effect size nor statistical significance establishes practical importance, causation, or performance beyond the observed runs.

Shape-based interpretation (rules of thumb) ​

  • Fast-start / early-plateau: high early checkpoint median + early plateau time + low late discovery share.
  • Steady: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
  • Slow-burn / late-surge: low early checkpoints but high late discovery share and later plateau time; often higher final median.

Limitations ​

  • Core count-based charts use normalized event identities stored under the legacy bugs_found column. They do not count crash inputs, but they are not confirmed root-cause bug counts.
  • Use known_bug_report.md, known_bug_summary.csv, and known_bug_findings.csv for evidence-backed known-bug hit rates; use broken_invariants.md / broken_invariants.csv for raw invariant identities.
  • Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
  • Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.

Broken invariants ​

  • Budget filter: 4.00h
  • Events considered: 116 / 116
  • Unique invariants: 16

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Per-fuzzer totals ​

FuzzerInvariants
echidna16
foundry14
medusa15
recon-fuzzer14

High-level overlap ​

  • Shared by all active fuzzers: 14
  • Exclusive to echidna: 1
  • Exclusive to foundry: 0
  • Exclusive to medusa: 0
  • Exclusive to recon-fuzzer: 0

Grouped invariants ​

Exclusive to echidna (1)
  • AssertionFailed
Exclusive to foundry (0)

None.

Exclusive to medusa (0)

None.

Exclusive to recon-fuzzer (0)

None.

Shared by all active fuzzers (14)
  • assert_canary
  • invariant_canary
  • testBurnBalance
  • testChangeSupply
  • testMintBalanceRounding
  • testOptInBalance
  • testOptInBalanceRounding
  • testRebasingCreditsPerTokenNotIncreased
  • testTotalSupplyLessThanTotalBalance
  • testTransferBalanceReceivedLessRounding
  • testTransferBalanceReceivedMore
  • testTransferBalanceSentLess
  • testTransferSelf
  • testTransferWithinBalanceDoesNotRevert

Top shared subsets (top 1 by size):

echidna, medusa (1)
  • testTransferFromShouldNotRevert

Ground-truth known-bug mapping ​

  • Target: origin-dollar
  • Evidence-pinned commit: 299ec6b6bf5401b43efc3e9f7ee5cab3e76167c4
  • Resolution: matched run repository and commit
  • Catalog: benchmarks/known_bugs.json

Known bugs are limited to handlers explicitly annotated with hasKnownIssue in the pinned target. Each handler remains a separate catalog entry because the target does not provide evidence that any pair is the same root cause.

Canaries are reported separately as harness-health checks and are never included in the known-bug hit-rate denominator.

Per-fuzzer results ​

FuzzerReplicatesKnown-bug replicate hitsCatalog coverageCanary replicate hitsUnmapped event findings
echidna211/24 (45.8%)6/12 (50.0%)4/4 (100.0%)15
foundry212/24 (50.0%)6/12 (50.0%)4/4 (100.0%)12
medusa212/24 (50.0%)6/12 (50.0%)4/4 (100.0%)14
recon-fuzzer212/24 (50.0%)6/12 (50.0%)4/4 (100.0%)12

Known-bug catalog ​

Ground-truth IDKindTitleEvidence
origin-dollar/change-supply-mismatchdocumented-known-issuechangeSupply does not exactly reach the requested total supplysource
origin-dollar/total-supply-below-balancesdocumented-known-issueTotal supply can fall below the sum of account balancessource
origin-dollar/non-rebasing-supply-above-totaldocumented-known-issueNon-rebasing supply can exceed total supplysource
origin-dollar/rebasing-credits-per-token-increasedocumented-known-issueRebasing credits per token can increasesource
origin-dollar/non-rebasing-supply-below-balancesdocumented-known-issueNon-rebasing supply can fall below non-rebasing balancessource
origin-dollar/mint-balance-shortfalldocumented-known-issueMinting can increase an account balance by less than the minted amountsource
origin-dollar/burn-balance-shortfalldocumented-known-issueBurning can decrease an account balance by less than the burned amountsource
origin-dollar/burn-all-leaves-balancedocumented-known-issueBurning an account's full balance can leave a nonzero balancesource
origin-dollar/burn-all-revertsdocumented-known-issueBurning an account's full balance can revertsource
origin-dollar/transfer-recipient-shortfalldocumented-known-issueA transfer recipient can receive less than the transferred amountsource
origin-dollar/transfer-sender-shortfalldocumented-known-issueA transfer sender's balance can decrease by less than the transferred amountsource
origin-dollar/transfer-within-balance-revertsdocumented-known-issueA transfer within the sender's balance can revertsource

Unmapped event identities ​

FuzzerEvent identityReplicates observed
echidnaAssertionFailed2
echidnatestMintBalanceRounding2
echidnatestOptInBalance2
echidnatestOptInBalanceRounding2
echidnatestTransferBalanceReceivedLessRounding2
echidnatestTransferBalanceReceivedMore2
echidnatestTransferFromShouldNotRevert1
echidnatestTransferSelf2
foundrytestMintBalanceRounding2
foundrytestOptInBalance2
foundrytestOptInBalanceRounding2
foundrytestTransferBalanceReceivedLessRounding2
foundrytestTransferBalanceReceivedMore2
foundrytestTransferSelf2
medusatestMintBalanceRounding2
medusatestOptInBalance2
medusatestOptInBalanceRounding2
medusatestTransferBalanceReceivedLessRounding2
medusatestTransferBalanceReceivedMore2
medusatestTransferFromShouldNotRevert2
medusatestTransferSelf2
recon-fuzzertestMintBalanceRounding2
recon-fuzzertestOptInBalance2
recon-fuzzertestOptInBalanceRounding2
recon-fuzzertestTransferBalanceReceivedLessRounding2
recon-fuzzertestTransferBalanceReceivedMore2
recon-fuzzertestTransferSelf2

Counting semantics ​

  • A canonical known-bug ID counts at most once per replicate, even when several event aliases or counterexamples reach it.
  • A replicate is identified by (run_id, instance_id, fuzzer).
  • Known-bug hit rate is canonical bug/replicate hits ÷ (cataloged bugs × replicates).
  • Unmapped rows are distinct normalized event identities per replicate. They are triage candidates, not claimed bugs.
  • Crash inputs and corpus files are not used as bug identities.

Runner resource usage ​

  • Budget filter: 4.00h
  • Instances with metrics: 8
  • Total samples: 22517

Per-fuzzer medians (across instances) ​

FuzzerInstancesCPU active avg (%)CPU active peak (%)Memory used avg (GiB)Memory used peak (GiB)Memory used avg (%)Memory used peak (%)
echidna223.9030.6517.8023.3258.1076.09
foundry299.90100.000.961.223.134.00
medusa286.7790.411.071.423.484.63
recon-fuzzer225.1548.881.261.834.125.97

Instance stats ​

InstanceFuzzerSamplesDuration (h)CPU active avg (%)CPU active peak (%)Memory avg (GiB)Memory peak (GiB)Memory avg (%)Memory peak (%)
i-0497450927a2a89a9-echidna-v2.3.1echidna28264.0023.8831.3017.4824.0457.0478.43
i-06b518158d3ba6fa8-echidna-v2.3.1echidna28274.0023.9230.0018.1322.6059.1573.75
i-024a9959ae72deb9f-foundry-git-02c05d9foundry27963.9899.90100.000.961.163.143.80
i-0eb2f56ba7c7e57aa-foundry-git-02c05d9foundry27953.9899.90100.000.961.293.124.19
i-04249f4c316c1db9e-medusa-v1.4.1medusa28094.0086.7391.811.051.493.424.85
i-0c374946c09e28ebd-medusa-v1.4.1medusa28074.0086.8189.011.091.353.554.41
i-00286b19f02049aa2-recon-v0.4.6recon-fuzzer28294.0025.1546.441.271.734.155.64
i-0922c2f8649781881-recon-v0.4.6recon-fuzzer28284.0025.1551.311.251.934.096.30

Manifest ​

  • scfuzzbench_commit: b204568bc0d8487e7e87f240a477c4b589868a97
  • target_repo_url: https://github.com/scfuzzbench/origin-dollar-scfuzzbench
  • target_commit: 299ec6b6bf5401b43efc3e9f7ee5cab3e76167c4
  • benchmark_type: property
  • instance_type: c6a.4xlarge
  • instances_per_fuzzer: 2
  • timeout_hours: 4
  • aws_region: us-east-1
  • ubuntu_ami_id: ami-052355af2a014bd2c
  • foundry_version: 1.7.2-dev
  • foundry_git_repo: https://github.com/foundry-rs/foundry
  • foundry_git_ref: 02c05d970d2801da0aef8b82486ce84b01ede36d
  • foundry_source_patch: scfuzzbench-throughput-progress-v1@sha256:2ee9e69b77c8007c78c816eb9ca791684aa5ecede0651b63f86cdd2e055eb17e
  • echidna_version: 2.3.1
  • medusa_version: 1.4.1
  • recon_version: 0.4.6
  • fuzzer_keys: echidna, foundry, medusa, recon-fuzzer

Artifacts ​

Fully static. Generated in CI from S3 run artifacts.

Copyright © Recon-Fuzz.