Generating minimal irreducible polynomials over GF(p)

[fredrik-johansson]

This PR adds nmod_poly_minimal_irreducible for deterministically generating irreducible polynomials with minimal weight (maximal sparsity) for any word-size prime p and degree n.

In general, there can be many irreducible polynomials of minimal weight for a given (p, n). The polynomial is chosen deterministically among these candidates as the smallest polynomial with respect to a certain lexicographic ordering of coefficients and exponents. The specific ordering is an arbitrary implementation detail, but does not depend on details of FLINT's RNG. It would not be hard to extend this function to allow iterating over all the minimal irreducible polynomials, if someone is interested in that.

We also add fq_nmod_ctx_init_minimal_weight_ui and change fq_nmod_ctx_init_ui to call this when a Conway polynomial is not available, replacing the previous behavior of generating a random polynomial (which would not always be maximally sparse). Some users may actually prefer the minimal weight moduli over Conway polynomials too as they allow faster arithmetic, but let's not touch that behavior for now.

Note: nmod_poly_minimal_irreducible implements various quick reducibility tests which could also be used to speed up nmod_poly_randtest_irreducible and nmod_poly_is_irreducible, but I'll leave that for the future.

Example improvement: constructing $\mathrm{GF}(5^{3125})$

• This PR: takes 5.1 seconds, returning x^3125 + x^90 + x^64 + 1
• Before: takes 80 seconds, returning x^3125 + 4 * x^3084 + 4 * x^3050 + 4 * x^3047 + 4 * x^3028 + x^3012 + 2 * x^3003 + 2 * x^2983 + 4 * x^2974 + x^2973 + x^2964 + x^2951 + x^2933 + x^2914 + x^2894 + 4 * x^2877 + x^2871 + 2 * x^2855 + 2 * x^2852 + 3 * x^2818 + 2 * x^2808 + x^2799 + 2 * x^2798 + 4 * x^2772 + 3 * x^2770 + 3 * x^2683 + 4 * x^2670 + 4 * x^2661 + 4 * x^2654 + x^2641 + x^2636 + 3 * x^2615 + 2 * x^2608 + 4 * x^2592 + 4 * x^2571 + 3 * x^2544 + 2 * x^2537 + x^2534 + x^2512 + 4 * x^2511 + 2 * x^2505 + 4 * x^2504 + 4 * x^2493 + x^2486 + 3 * x^2480 + 3 * x^2468 + 2 * x^2465 + x^2459 + 3 * x^2439 + 4 * x^2431 + 2 * x^2428 + x^2415 + 2 * x^2389 + 3 * x^2383 + x^2377 + 4 * x^2357 + x^2350 + 3 * x^2345 + x^2342 + 2 * x^2318 + 3 * x^2280 + 2 * x^2278 + 3 * x^2275 + x^2258 + 4 * x^2155 + 4 * x^2148 + x^2147 + 4 * x^2103 + 2 * x^2101 + x^2083 + 4 * x^2066 + 3 * x^2063 + x^2058 + 3 * x^2017 + 3 * x^1998 + 2 * x^1995 + 3 * x^1992 + 2 * x^1987 + 3 * x^1975 + 2 * x^1972 + 2 * x^1954 + x^1948 + 2 * x^1920 + x^1919 + 2 * x^1884 + 2 * x^1873 + 2 * x^1857 + 3 * x^1853 + 4 * x^1848 + 2 * x^1847 + 3 * x^1846 + 2 * x^1843 + x^1841 + 3 * x^1840 + x^1822 + x^1819 + 4 * x^1815 + 4 * x^1812 + x^1786 + 3 * x^1777 + 2 * x^1774 + 4 * x^1771 + 3 * x^1749 + 4 * x^1747 + 4 * x^1745 + 4 * x^1737 + 3 * x^1735 + x^1733 + 2 * x^1708 + x^1706 + 3 * x^1689 + 3 * x^1686 + 2 * x^1668 + x^1666 + x^1663 + 4 * x^1624 + 3 * x^1616 + 3 * x^1613 + 2 * x^1592 + 3 * x^1569 + 4 * x^1556 + x^1521 + 2 * x^1501 + x^1495 + x^1487 + 4 * x^1457 + 4 * x^1454 + 4 * x^1450 + 4 * x^1446 + 4 * x^1422 + 3 * x^1421 + 2 * x^1402 + 4 * x^1388 + 4 * x^1384 + 2 * x^1377 + 2 * x^1367 + 4 * x^1358 + 4 * x^1357 + 2 * x^1346 + 2 * x^1336 + x^1314 + x^1298 + x^1294 + 4 * x^1258 + 3 * x^1253 + 3 * x^1225 + 2 * x^1213 + 4 * x^1205 + x^1179 + 4 * x^1169 + 4 * x^1155 + 4 * x^1139 + x^1123 + 3 * x^1117 + 3 * x^1112 + x^1093 + 3 * x^1083 + 4 * x^1075 + x^1073 + 3 * x^1063 + x^1051 + 2 * x^1050 + 2 * x^1043 + 2 * x^1039 + 3 * x^1038 + x^1036 + 4 * x^1026 + x^1009 + x^1006 + 2 * x^992 + 4 * x^979 + 2 * x^973 + 3 * x^964 + x^947 + 4 * x^944 + 3 * x^928 + 2 * x^908 + 4 * x^906 + 2 * x^903 + 2 * x^875 + x^871 + 4 * x^803 + 3 * x^786 + 4 * x^761 + 2 * x^741 + 3 * x^708 + 2 * x^701 + x^700 + 2 * x^699 + x^691 + 3 * x^646 + 4 * x^637 + x^624 + 2 * x^603 + 4 * x^593 + x^573 + x^562 + x^555 + 3 * x^554 + 2 * x^546 + x^543 + 4 * x^507 + 4 * x^494 + 4 * x^479 + x^473 + 4 * x^440 + 2 * x^435 + 3 * x^420 + 4 * x^418 + 3 * x^417 + 4 * x^393 + 2 * x^379 + x^376 + 4 * x^371 + 3 * x^369 + 3 * x^348 + x^335 + x^312 + 2 * x^301 + 4 * x^296 + 2 * x^291 + 2 * x^284 + 4 * x^259 + x^250 + x^219 + 3 * x^201 + x^179 + 4 * x^176 + 2 * x^163 + 2 * x^143 + 2 * x^133 + x^112 + 2 * x^102 + x^85 + 4 * x^73 + x^59 + 3 * x^54 + 3 * x^46 + 3 * x^38 + x^16 + 4 * x^11 + x^10 + 2 * x^7 + 2
• Flint-3.3: takes 309 seconds, returning the same ugly polynomial as above

Constructing $\mathrm{GF}(5^{4096})$ (special case, minimal irreducible is a binomial):

• This PR: 0.00015 seconds, returning x^4096 + 2
• Before: 3.0 seconds, returning x^4096 + 2
• Flint-3.3: 6.9 seconds, returning x^4096 + 2

Constructing $\mathrm{GF}(2^n)$ for all $1 \le n \le 1000$:

• This PR: 67.6 seconds
• Before: 451.8 seconds
• Flint-3.3: 1054 seconds

Constructing $\mathrm{GF}(3^n)$ for all $1 \le n \le 1000$:

• This PR: 41.8 seconds
• Before: 497.9 seconds
• Flint-3.3: 1440 seconds

Constructing $\mathrm{GF}(5^n)$ for all $1 \le n \le 1000$:

• This PR: 71.9 seconds
• Before: 643.1 seconds
• Flint-3.3: 2238 seconds

Constructing $\mathrm{GF}(123457^n)$ for all $1 \le n \le 500$:

• This PR: 184.3 seconds
• Before: 294.6 seconds
• Flint-3.3: 545.5 seconds

Constructing $\mathrm{GF}(1152921504606847009^n)$ for all $1 \le n \le 200$:

• This PR: 17.3 seconds
• Before: 28.5 seconds
• Flint-3.3: 69.7 seconds

[fredrik-johansson]

@fieker @thofma Let me know if this looks fast enough to consider Oscar #5358. solved, or if you have other ideas to improve this.

I've also started compiling some lookup tables at https://github.com/fredrik-johansson/irreducible-polynomials.

We could easily add some of these to FLINT to complement the existing Conway tables, it's just a question of balancing utility vs bloat. If one wants instant lookups for GF(p^n) for n <= 10000 say, I guess we would need something like 50 KB per prime with tables in a compact binary form.

[thofma]

Great improvement! I would be happy with it. No need for precomputed tables for me.

For fun, I tried pari/gp and it is also instant for 5^(5^6). I think it uses Artin-Schreier theory that you mentioned before.

? c = ffgen(5^(5^5 - 1), 'c); c.mod;
time = 202 ms.
? c = ffgen(5^(5^5), 'c); c.mod;
time = 55 ms.

I am still wondering how Magma does it:

> time GF(5^3125);
Finite field of size 5^3125
Time: 0.010

> f := x^3125 + x^7 + x^4 + 4*x^3 + 4*x^2 + 4*x + 1; // this is the defining polynomial for GF(5^3125)
> time IsIrreducible(f);
true
Time: 0.140

[fredrik-johansson]

nmod_poly_is_irreducible takes 0.38 seconds for the degree 3125 FLINT polynomial on my machine and 0.53 seconds for the Magma polynomial. That's in the ballpark at least. I'm pretty sure we can do polynomial arithmetic over GF(5) quite a bit faster than what we do currently, but I wonder if there's some other trick missing in the irreducibility test.

[edgarcosta]

From: Basic rings -> Finite fields in the handbook:

A database of low-term irreducible polynomials for finite fields of small characteristic is available in Magma (see the function IrreducibleLowTermGF2Polynomial below). This database goes up to degree 120,000 for GF(2), and to reasonably high degrees for the fields GF(q) for q up to 128. Thus one can create the finite field GF(pk) for p, k within this range and compute within the field without any delay in the creation. Advantage is also taken of the special form of the defining polynomial.

I wonder if there's some other trick missing in the irreducibility test.

@AllanKSteel would be right person to comment on what the generic irreducible test is doing in Magma.