Hi Everyone,
It’s been a while when I sent my email about introducing an automatic benchmark suite I had been working: https://externals.io/message/116323#116323 Now I would like to share an
important milestone of this project:
After doing lots of improvements and fixes throughout the year, the suite now reached the
required maturity level to be able to use it within PHP’s nightly CI jobs (experimentally for now).
Previously, consistency and reliability issues, as well as the lack of financial support prevented
us from doing so. As these issues were addressed by now, we will be able to track the
performance changes of PHP, starting with 8.5.
The initial results are very promising for real-word app based tests (Laravel, Symfony,
Wordpress), indicating that the results are consistent and seemingly reliable. For example run to
run variation of the median absolute results typically don’t exceed 0.001 sec. When
benchmarking a php-src commit against itself, the relative difference for each real-word test
varied between 0.01% and 0.06% (while 0.0% would have been the best possible value).
Unfortunately, unlike real-word apps, micro benchmarks (bench.php and micro_bench.php)
sometimes give inconsistent results, so there is still room for improvements. Besides making
micro benchmarks more reliable, future plans include adding visualization, and coming up with
some notification system which tells us if there is a notable performance regression or progression.
For anyone interested in the project and the results, the benchmark suite is available at
https://github.com/kocsismate/php-version-benchmarks, while the nightly results are committed
into https://github.com/php/real-time-benchmark-data in both human friendly (markdown) and in
machine processable format (tsv). Additionally, a more detailed explanation about the benchmark’s methodology can be found at https://github.com/php/php-src/pull/15696.
Regards,
Máté