R version used for performance comparison too old

[szhorvat]

The performance comparison shown on http://julialang.org/ indicates that R version 2.9.0 was used. This is a 3 year old (!) version of R. The current version is 2.14.2 which has a byte compiler which speeds it up considerable.

I believe that for a comparison to be useful at all, it should use recent releases of the systems it compares. Of course a newer version of R most likely won't beat Julia either, but it seems kind of pointless to include R at all in the comparison if the version that is used is so terribly outdated, especially considering the recent performance improvements R has received.

[ViralBShah]

I wasn't aware that the current version of R has a bytecode compiler. That's pretty amazing, and will certainly help. In any case the perf benchmark should be run with the latest version of R. The version numbering of R seems to be a bit strange, because at first glance one would not expect there to be much difference between 2.9 and 2.14.

[StefanKarpinski]

Ok, I can download the new version and re-run those benchmarks.

[StefanKarpinski]

So R is doing better now, using version 2.14.2 — it's 2-3 orders of magnitude slower than C++ instead of 3-4. That is a considerable speedup. Unfortunately, it also got more than three times slower at doing matrix multiplication.

[ViralBShah]

Javascript rand_mat_mul seems to have improved thanks to the patches contributed.

Can we close this one?

-viral

On 05-Mar-2012, at 12:13 AM, Stefan Karpinski wrote:

So R is doing better now, using version 2.14.2 — it's 2-3 orders of magnitude slower than C++ instead of 3-4. That is a considerable speedup. Unfortunately, it also got more than three times slower at doing matrix multiplication.

---

Reply to this email directly or view it on GitHub:
#514 (comment)

[szhorvat]

Well, there's also the Revolution Analytics version of R which should be much faster at matrix multiplication, but it is not free for commercial use (the Community and the Academic edition are free).,

[StefanKarpinski]

I think we'll stick with the standard distribution R.

[ViralBShah]

Yes, I think it is best to use the standard distribution, since that is often what most people are likely to use. In any case, people will know that they need to use a better BLAS to get stuff like rand_mat_mul to work fast.

[StefanKarpinski]

I really doubt this is due to their BLAS being bad. It must be overhead introduced before calling the BLAS.

[ViralBShah]

Szabolcs Horvat pointed out that we were using element-wise power rather than matrix power in python's randmatstat. I have committed a fix - but we will need to re-run the tests.

-viral

On 05-Mar-2012, at 12:25 AM, Stefan Karpinski wrote:

I really doubt this is due to their BLAS being bad. It must be overhead introduced before calling the BLAS.

---

Reply to this email directly or view it on GitHub:
#514 (comment)

[StefanKarpinski]

This is done — didn't change the benchmark appreciably. I guess the matrices are small enough that a real matrix power is about the same amount of work as an element-wise power.

[jiahao]

I've run the microbenchmarks against standard R and Revolution R. (x-ref julia-users)

The only two benchmarks that differ significantly are fib and rand_mat_mul.

	Fortran gcc-4.8.2	Julia 0.4-dev+2241	R 3.1.2	RRO 8.0.1
fib	0.49	2.01	519.81	466.03
parse_int	4.08	1.58	49.34	49.34
quicksort	1.12	1.02	231.82	231.82
mandel	0.73	0.72	54.06	56.51
pi_sum	0.84	0.83	13.47	14.45
rand_mat_stat	0.99	1.57	15.82	14.46
rand_mat_mul	4.59	1.03	6.13	1.85