Skip to content

Conversation

EwoutH
Copy link
Member

@EwoutH EwoutH commented Aug 28, 2024

BoltzmannWealth model to the benchmarks. It's probably the "best" test of Mesa that we have, since it does a lot of Mesa things and not a lot of other things.

A separate PR will update it to use AgentSet functionality, but I want to see the performance difference between that. This PR just adds the example model from mesa-examples.


@quaquel I noticed in that with the addition of discrete event scheduling in #2066 all models were converted to DEVS. While I understand it was useful to test, and I think it's great to have that in the benchmark for one or two models, I think for all is a bit overkill as long as it's experimental.

So the novelty is clearly in WolfSheep, I would propose keeping that on DEVS and reverting the other two back to our regular activation.

BoltzmannWealth model to the benchmarks
@EwoutH EwoutH added testing Release notes label ci Release notes label labels Aug 28, 2024
@EwoutH EwoutH requested a review from quaquel August 28, 2024 09:12
@EwoutH EwoutH marked this pull request as ready for review August 28, 2024 09:12
Copy link

Performance benchmarks:

Model Size Init time [95% CI] Run time [95% CI]
Schelling small 🟢 -14.8% [-15.3%, -14.4%] 🟢 -20.7% [-20.9%, -20.5%]
Schelling large 🟢 -15.0% [-15.6%, -14.4%] 🟢 -17.8% [-18.0%, -17.5%]
WolfSheep small 🟢 -16.1% [-17.1%, -15.0%] 🟢 -17.0% [-17.3%, -16.8%]
WolfSheep large 🟢 -15.4% [-15.6%, -15.2%] 🟢 -16.5% [-16.7%, -16.3%]
BoidFlockers small 🟢 -14.2% [-15.0%, -13.4%] 🟢 -9.8% [-10.5%, -9.2%]
BoidFlockers large 🟢 -14.4% [-15.0%, -13.8%] 🟢 -12.2% [-13.0%, -11.5%]

Copy link

Performance benchmarks:

Model Size Init time [95% CI] Run time [95% CI]
Schelling small 🟢 -13.7% [-14.2%, -13.2%] 🟢 -21.5% [-21.7%, -21.3%]
Schelling large 🟢 -14.8% [-15.6%, -14.0%] 🟢 -11.9% [-12.7%, -10.9%]
WolfSheep small 🟢 -13.9% [-15.0%, -12.6%] 🔵 +1.5% [-0.8%, +3.9%]
WolfSheep large 🟢 -13.9% [-14.5%, -13.3%] 🟢 -13.6% [-15.2%, -11.8%]
BoidFlockers small 🟢 -13.2% [-13.8%, -12.5%] 🟢 -9.4% [-10.2%, -8.6%]
BoidFlockers large 🟢 -14.5% [-15.1%, -14.0%] 🟢 -12.9% [-13.5%, -12.4%]

@quaquel
Copy link
Member

quaquel commented Aug 28, 2024

Good idea.

@quaquel I noticed in that with the addition of discrete event scheduling in #2066 all models were converted to DEVS. While I understand it was useful to test, and I think it's great to have that in the benchmark for one or two models, I think for all is a bit overkill as long as it's experimental.

So the novelty is clearly in WolfSheep, I would propose keeping that on DEVS and reverting the other two back to our regular activation.

Fine with me. Do we also want to have a mix of scheduler use and agentset use?

@EwoutH
Copy link
Member Author

EwoutH commented Aug 28, 2024

Thanks, and yes.

Can you approve? Merging is blocked until.

@quaquel quaquel merged commit 3e25d9c into projectmesa:main Aug 28, 2024
@quaquel quaquel deleted the benchmark_Boltzman branch August 28, 2024 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Release notes label testing Release notes label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants