scfuzzbench

Appearance

Run gh-29781264950-1 ​

Trial run

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Charts ​

Bugs Over TimeTime To KFinal DistributionPlateau And Late ShareInvariant Overlap (UpSet)Coverage Over TimeCorpus Size Over TimeSeq/s Over TimeTx/s Over TimeGas/s Over TimeCPU Usage Over TimeMemory Usage Over Time

Report ​

Fuzzer Benchmark Report (from bug-count CSV) ​

  • Time budget: 4.00h

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Executive summary ​

This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

Bugs found at fixed time budgets (median [IQR]) ​

FuzzerRuns1h4h
medusa214 [14,14]14 [14,14]
echidna214 [14,14]14 [14,15]
foundry212 [12,12]14 [13,14]
recon-fuzzer26 [6,6]6 [6,6]

Overall metrics ​

FuzzerAUC (norm)Plateau timeLate discovery shareFinal medianFinal IQR
medusa0.8980.40h0.000140.00
echidna0.8891.30h0.000140.50
foundry0.8252.00h0.000140.50
recon-fuzzer0.3920.10h0.00060.00

Milestones: time-to-k and success rates ​

Fuzzertime-to-1 (p50)time-to-3 (p50)time-to-5 (p50)reach-1 ratereach-3 ratereach-5 rate
medusa0.00h0.10h0.10h100.0%100.0%100.0%
echidna0.10h0.10h0.10h100.0%100.0%100.0%
foundry0.10h0.10h0.10h100.0%100.0%100.0%
recon-fuzzer0.00h0.10h0.10h100.0%100.0%100.0%

Throughput metrics (if supported by log format) ​

Values are run-level rates aggregated per fuzzer; n/a indicates the parser could not recover that metric from logs.

FuzzerRunsTx/s runsTx/s p50 [p25,p75]Gas/s runsGas/s p50 [p25,p75]
medusa224848.00 [4761.50,4934.50]21483285370.00 [1468525720.00,1498045020.00]
echidna22793.35 [778.10,808.60]29723997054.00 [9039376017.00,10408618091.00]
foundry2225930.58 [23388.79,28472.38]26378039431.01 [6212524989.01,6543553873.01]
recon-fuzzer22604.70 [585.05,624.35]288277160.50 [80386847.75,96167473.25]

Progress metrics from logs (fuzzer-specific proxies) ​

Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.

FuzzerRunsSeq/s runsSeq/s p50 [p25,p75]Corpus runsCorpus p50 [p25,p75]
medusa2248.50 [47.75,49.25]2139.00 [136.00,142.00]
echidna20n/a262.00 [61.50,62.50]
foundry20n/a20.00 [0.00,0.00]
recon-fuzzer20n/a284.00 [71.00,97.00]

Coverage over time (opt-in, fuzzer-native signals) ​

  • Status: enabled
  • Source-based coverage is preferred, but the current timestamped progress formats expose only native bytecode/instrumentation proxies. The signals below have different units and must not be ranked, pooled, or plotted on a shared scale.
  • Foundry showmap artifacts are a separate, opt-in post-campaign edge replay; they are not real-time source coverage.
FuzzerNative signalSource-based?Live availabilityRuns with signalTemporal observationsObserved parser provenanceFinal native signal p50 [p25,p75]Limitation
medusabranches hit (branches)No (runtime proxy)Medusa status lines2/2live; 9574 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budgettext-metrics:branches_hit5204.00 [5203.50,5204.50]Branch identities depend on compiled bytecode/config; no source locations.
echidnacov coverage points (coverage points)No (runtime proxy)Echidna status lines2/2live; 9567 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budgettext-metrics:cov44530.50 [44528.25,44532.75]Tool/config-specific counter; no source locations in the progress log.
foundrycumulative_edges_seen (instrumented edges)No (runtime proxy)Compatible Foundry JSON pulse builds2/2live; 5731 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budgetjson-metrics:cumulative_edges_seen0.00 [0.00,0.00]Edge identities depend on the build/instrumentation; showmap replay is separate and post-campaign.
recon-fuzzerUnique instructions (instructions)No (runtime proxy)End-of-run Recon summary (not a live series)2/2end-of-run only; 2 sample(s); excluded from time-series charttext-metrics:unique_instructions45237.00 [45228.50,45245.50]Tool/version-specific final instruction count; no source locations or intermediate observations.

Function selector sanity checks ​

Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in selector_distribution.csv; selector_summary.json preserves statuses and provenance.

  • Expected-set status: available (67 selector(s))
  • Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth
FuzzerStatusInstances with telemetrySaved-corpus callsUnique selectors
echidnaavailable2/21232968
foundryunavailable0/200
medusaavailable2/21074467
recon-fuzzeravailable2/21612867
Most common saved-corpus selectors ​
FuzzerSelectorFunctionCallsShare
echidna0x0325f4e1addSplitsReceiver(uint8,uint8,uint32)2401.95%
echidna0xed0c06d6removeAllSplits(uint8)2321.88%
echidna0x81b0058ctestSqueezeViewVsActual(uint8,uint8)2311.87%
echidna0xb89fedbcaddStreamWithClamping(uint8,uint8,uint160,uint32,uint32,int128)2271.84%
echidna0xfabaa91dtestSetStreamBalanceWithdrawAllShouldNotRevert(uint8)2151.74%
echidna0x82ce0f6dtestReceiveStreamsShouldNotRevert(uint8)2101.70%
echidna0x8319b8dbreceiveStreamsSplitAndCollectToSelf(uint8)2101.70%
echidna0x45b9b0batestReceiveStreamsViewVsActual(uint8,uint32)2081.69%
echidna0x79f368d0squeezeAllSenders(uint8)2031.65%
echidna0x1a3ed582setStreams(uint8,uint8,uint160,uint32,uint32,int128)2011.63%
medusa0x0325f4e1addSplitsReceiver(uint8,uint8,uint32)3493.25%
medusa0x1ba3a23ctestReceiveStreamsViewConsistency(uint8,uint32)3062.85%
medusa0x8d8b7f3bsetSplits(uint8,uint8,uint32)2932.73%
medusa0x462664e4testRemoveStreamShouldNotRevert(uint8,uint256)2852.65%
medusa0x218be192invariantSumAmtDeltaIsZero(uint8)2802.61%
medusa0x1a3ed582setStreams(uint8,uint8,uint160,uint32,uint32,int128)2792.60%
medusa0x47bc779btestAddSplitsShouldNotRevert(uint8,uint8,uint32)2682.49%
medusa0x3bfbc3bdsetStreamBalance(uint8,int128)2612.43%
medusa0xec13b1catestAddStreamShouldNotRevert(uint8,uint8,uint160,uint32,uint32,int128)2532.35%
medusa0x2e0c28fbsetMaxEndHints(uint32,uint32)2482.31%
recon-fuzzer0xbcb054fdaddStreamImmediatelySqueezable(uint8,uint8,uint160)3242.01%
recon-fuzzer0xe2b3d196splitAndCollectToSelf(uint8)3242.01%
recon-fuzzer0x92da0f7atestSqueezeWithFuzzedHistoryShouldNotRevert(uint8,uint8,uint256,bytes32)3201.98%
recon-fuzzer0xed0c06d6removeAllSplits(uint8)3181.97%
recon-fuzzer0x2b95b898invariantWithdrawAllTokensShouldNotRevert()3141.95%
recon-fuzzer0x662fe867invariantAmtPerSecVsMinAmtPerSec(uint8,uint256)3121.93%
recon-fuzzer0xb777536fsqueezeToSelf(uint8)3091.92%
recon-fuzzer0x19ca05b3testSplittableAfterSplit(uint8)2971.84%
recon-fuzzer0x13a600d0addStream(uint8,uint8,uint160,uint32,uint32,int128)2851.77%
recon-fuzzer0x218be192invariantSumAmtDeltaIsZero(uint8)2841.76%
Selector diagnostic findings ​
  • INFO: i-093a6dd5d2b926065-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0x21d76db7
  • INFO: i-0ff08ba021a054554-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0x21d76db7
Selector telemetry limitations ​
  • i-0064e4a60c38595e5-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution
  • i-058a92fa40dc71e39-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution

Statistical comparison (Mann-Whitney U and Vargha-Delaney A12) ​

Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts. The tests are two-sided, with Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

Warnings:

  • One or more fuzzers have fewer than 5 runs. Statistical power may be limited.
Fuzzer AFuzzer BMedian AMedian BU statisticA12 (A over B)Effect magnitudep-valuep (corrected)Significant
medusaechidna1414.51.00.250large0.61711.0000no
medusafoundry1413.53.00.750large0.61711.0000no
medusarecon-fuzzer1464.01.000large0.19391.0000no
echidnafoundry14.513.53.50.875large0.41421.0000no
echidnarecon-fuzzer14.564.01.000large0.22071.0000no
foundryrecon-fuzzer13.564.01.000large0.22071.0000no

No pairwise comparison reached significance after Bonferroni correction.

Note: A12 is the probability that a randomly selected Fuzzer A run has a higher final bug count than a randomly selected Fuzzer B run, counting ties as half. Values above 0.5 favor A, values below 0.5 favor B, and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5 are negligible (<0.06), small (<0.14), medium (<0.21), and large (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power. A12 and its magnitude label are descriptive; the thresholds are rules of thumb. Neither the effect size nor statistical significance establishes practical importance, causation, or performance beyond the observed runs.

Shape-based interpretation (rules of thumb) ​

  • Fast-start / early-plateau: high early checkpoint median + early plateau time + low late discovery share.
  • Steady: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
  • Slow-burn / late-surge: low early checkpoints but high late discovery share and later plateau time; often higher final median.

Limitations ​

  • Core count-based charts use normalized event identities stored under the legacy bugs_found column. They do not count crash inputs, but they are not confirmed root-cause bug counts.
  • Use known_bug_report.md, known_bug_summary.csv, and known_bug_findings.csv for evidence-backed known-bug hit rates; use broken_invariants.md / broken_invariants.csv for raw invariant identities.
  • Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
  • Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.

Broken invariants ​

  • Budget filter: 4.00h
  • Events considered: 96 / 96
  • Unique invariants: 16

Warning — trial run. This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

Per-fuzzer totals ​

FuzzerInvariants
echidna15
foundry14
medusa14
recon-fuzzer6

High-level overlap ​

  • Shared by all active fuzzers: 6
  • Exclusive to echidna: 1
  • Exclusive to foundry: 1
  • Exclusive to medusa: 0
  • Exclusive to recon-fuzzer: 0

Grouped invariants ​

Exclusive to echidna (1)
  • AssertionFailed
Exclusive to foundry (1)
  • testBalanceAtInFuture
Exclusive to medusa (0)

None.

Exclusive to recon-fuzzer (0)

None.

Shared by all active fuzzers (6)
  • assert_canary
  • invariantWithdrawAllTokensShouldNotRevert
  • invariant_canary
  • testReceiveStreamsShouldNotRevert
  • testSqueezableAmountCantBeUndone
  • testSqueezableAmountCantBeWithdrawn

Top shared subsets (top 2 by size):

echidna, foundry, medusa (7)
  • invariantAccountingVsTokenBalance
  • testSetStreamBalanceShouldNotRevert
  • testSetStreamBalanceWithdrawAllShouldNotRevert
  • testSetStreamsShouldNotRevert
  • testSqueezableVsReceived
  • testSqueezeWithDefaultHistoryShouldNotRevert
  • testSqueezeWithFuzzedHistoryShouldNotRevert
echidna, medusa (1)
  • testAddStreamShouldNotRevert

Ground-truth known-bug mapping ​

  • Target: drips
  • Evidence-pinned commit: c50d160ead6bf82b1d1071e853dc74da0b23f595
  • Resolution: matched run repository and commit
  • Catalog: benchmarks/known_bugs.json

The one real known bug is the pinned historical squeeze regression. Two assertion handlers are aliases for the same root cause and therefore count as one bug per run.

Canaries are reported separately as harness-health checks and are never included in the known-bug hit-rate denominator.

Per-fuzzer results ​

FuzzerReplicatesKnown-bug replicate hitsCatalog coverageCanary replicate hitsUnmapped event findings
echidna22/2 (100.0%)1/1 (100.0%)4/4 (100.0%)21
foundry22/2 (100.0%)1/1 (100.0%)4/4 (100.0%)19
medusa22/2 (100.0%)1/1 (100.0%)4/4 (100.0%)20
recon-fuzzer22/2 (100.0%)1/1 (100.0%)4/4 (100.0%)4

Known-bug catalog ​

Ground-truth IDKindTitleEvidence
drips/pre-update-time-squeezehistorical-regressionPreviously streamed funds can be removed from the squeezable balancesource

Unmapped event identities ​

FuzzerEvent identityReplicates observed
echidnaAssertionFailed2
echidnainvariantAccountingVsTokenBalance2
echidnainvariantWithdrawAllTokensShouldNotRevert2
echidnatestAddStreamShouldNotRevert2
echidnatestReceiveStreamsShouldNotRevert2
echidnatestSetStreamBalanceShouldNotRevert1
echidnatestSetStreamBalanceWithdrawAllShouldNotRevert2
echidnatestSetStreamsShouldNotRevert2
echidnatestSqueezableVsReceived2
echidnatestSqueezeWithDefaultHistoryShouldNotRevert2
echidnatestSqueezeWithFuzzedHistoryShouldNotRevert2
foundryinvariantAccountingVsTokenBalance2
foundryinvariantWithdrawAllTokensShouldNotRevert2
foundrytestBalanceAtInFuture2
foundrytestReceiveStreamsShouldNotRevert2
foundrytestSetStreamBalanceShouldNotRevert1
foundrytestSetStreamBalanceWithdrawAllShouldNotRevert2
foundrytestSetStreamsShouldNotRevert2
foundrytestSqueezableVsReceived2
foundrytestSqueezeWithDefaultHistoryShouldNotRevert2
foundrytestSqueezeWithFuzzedHistoryShouldNotRevert2
medusainvariantAccountingVsTokenBalance2
medusainvariantWithdrawAllTokensShouldNotRevert2
medusatestAddStreamShouldNotRevert2
medusatestReceiveStreamsShouldNotRevert2
medusatestSetStreamBalanceShouldNotRevert2
medusatestSetStreamBalanceWithdrawAllShouldNotRevert2
medusatestSetStreamsShouldNotRevert2
medusatestSqueezableVsReceived2
medusatestSqueezeWithDefaultHistoryShouldNotRevert2
medusatestSqueezeWithFuzzedHistoryShouldNotRevert2
recon-fuzzerinvariantWithdrawAllTokensShouldNotRevert2
recon-fuzzertestReceiveStreamsShouldNotRevert2

Counting semantics ​

  • A canonical known-bug ID counts at most once per replicate, even when several event aliases or counterexamples reach it.
  • A replicate is identified by (run_id, instance_id, fuzzer).
  • Known-bug hit rate is canonical bug/replicate hits ÷ (cataloged bugs × replicates).
  • Unmapped rows are distinct normalized event identities per replicate. They are triage candidates, not claimed bugs.
  • Crash inputs and corpus files are not used as bug identities.

Runner resource usage ​

  • Budget filter: 4.00h
  • Instances with metrics: 8
  • Total samples: 22374

Per-fuzzer medians (across instances) ​

FuzzerInstancesCPU active avg (%)CPU active peak (%)Memory used avg (GiB)Memory used peak (GiB)Memory used avg (%)Memory used peak (%)
echidna296.3598.1420.9623.2068.4075.71
foundry299.91100.001.001.313.254.27
medusa298.2798.991.301.604.265.21
recon-fuzzer296.1598.583.013.309.8210.78

Instance stats ​

InstanceFuzzerSamplesDuration (h)CPU active avg (%)CPU active peak (%)Memory avg (GiB)Memory peak (GiB)Memory avg (%)Memory peak (%)
i-08aeb92c1b17db4a3-echidna-v2.3.1echidna27744.0096.4898.2120.8623.0468.0875.17
i-095187848c5450d18-echidna-v2.3.1echidna27714.0096.2198.0721.0623.3768.7276.25
i-0064e4a60c38595e5-foundry-git-02c05d9foundry27973.9899.91100.000.991.303.254.25
i-058a92fa40dc71e39-foundry-git-02c05d9foundry27983.9999.91100.001.001.323.254.30
i-0ce1467fd9f8849bd-medusa-v1.4.1medusa28114.0098.1998.891.291.604.205.22
i-0fb580c2fdda2427b-medusa-v1.4.1medusa28114.0098.3499.091.321.594.325.19
i-093a6dd5d2b926065-recon-v0.4.6recon-fuzzer28064.0096.0798.493.033.339.8810.87
i-0ff08ba021a054554-recon-v0.4.6recon-fuzzer28064.0096.2398.662.993.289.7510.69

Manifest ​

  • scfuzzbench_commit: 6e0a6597390e5394ea1624b7d0889bf9addfd06e
  • target_repo_url: https://github.com/scfuzzbench/drips-fuzzing-scfuzzbench
  • target_commit: c50d160ead6bf82b1d1071e853dc74da0b23f595
  • benchmark_type: property
  • instance_type: c6a.4xlarge
  • instances_per_fuzzer: 2
  • timeout_hours: 4
  • aws_region: us-east-1
  • ubuntu_ami_id: ami-052355af2a014bd2c
  • foundry_version: 1.7.2-dev
  • foundry_git_repo: https://github.com/foundry-rs/foundry
  • foundry_git_ref: 02c05d970d2801da0aef8b82486ce84b01ede36d
  • foundry_source_patch: scfuzzbench-throughput-progress-v1@sha256:2ee9e69b77c8007c78c816eb9ca791684aa5ecede0651b63f86cdd2e055eb17e
  • echidna_version: 2.3.1
  • medusa_version: 1.4.1
  • recon_version: 0.4.6
  • fuzzer_keys: echidna, foundry, medusa, recon-fuzzer

Artifacts ​

Fully static. Generated in CI from S3 run artifacts.

Copyright © Recon-Fuzz.