Heap-buffer-overflow on intersection

[gabe-sherman]

I am an undergraduate student exploring automatic fuzzing harness generation for open source api's. I found that executing geosop instrumented with asan with these arguments:
./geosop -a "GEOMETRYCOLLECTION(POINT(0 0 59.083333333333336), POINT(1 1 59.083333333333336), LINESTRING(2 4 111 1, 2 223 22 2), POLYGON((42 -2.222222223222222e+155 111 1, 2 3 1 1, 2 4 111 NaN, 42 -2.222222223222222e+155 111 NaN)), POLYGON((2.2222222212222224e+149 4 111 1, 42 -2.222222222222222e+149 22 2, 2 4 111 1, 22 2222222222222223 22 NaN, 2.2222222212222224e+149 4 111 NaN)))" -b "LINESTRING (1.6409301755752343e+149 -5.298190649085575e+148, 1.6666666660752404e+149 -5.55555555396982e+148)" intersection

results in a heap-buffer-overflow.

Test Environment
Ubuntu 22.04, 64bit
Using the Latest version: 0aef713

[JamesParrott]

It would be good to prevent overflows, but there's also an argument that this use case is not supported.

Firstly:
Is there a need for NaN in computational geometry(*)? NaN (DoubleNotANumber) is actually used internally to initialise some shapes.

Secondly
Do any users need to define quantities, both to a precision of 1 unit, and also describe negative values as large as -2.2E+155 ?

The Planck length is 10^-35m, and the size of the observable universe is 10^27m.

Internally, strtod is used in the tokenizer. The WKT parser functions return doubles. If 64 bit doubles are available, they should support numbers as large as 10^308. So Gabe, if you haven't found the limits of floating point precision (or of your test machine), there's a possibility something more interesting is causing the error. But this is an extreme edge case.

NaN

(*) There is code in Geos to parse NaN (and even infinity). However, the libgeos WKT spec doesn't give any examples of NaN, and everything boils down to <Number>. NaN is a valid value according to some floating point standards, but it's entirely reasonable to conclude that by definition (the clue is in the name): NaN is not a <Number>.

https://libgeos.org/specifications/wkt/

The (OG) OGC WKT spec (7.2.1 - 7.2.2) excludes NaN:

<point> ::= <x> <y>
<x> ::= <signed numeric literal>
<signed numeric literal> ::= {<sign>}<unsigned numeric literal>
<unsigned numeric literal> ::= <exact numeric literal>|<approximate numeric literal>

<exact numeric literal> ::= <unsigned integer>
{<decimal point>{<unsigned integer>}}
|<decimal point><unsigned integer>

<approximate numeric
literal> ::=
<mantissa>E<exponent>

<mantissa> ::= <exact numeric literal>
<exponent> ::= <signed integer>
<exact numeric literal> ::= <unsigned integer>
{<decimal point>{<unsigned integer>}}
|<decimal point><unsigned integer>
<signed integer> ::= {<sign>}<unsigned integer>
<unsigned integer> ::= (<digit>)*

<digit> ::= 0|1|2|3|4|5|6|7|8|9

https://www.ogc.org/standard/sfa/
https://portal.ogc.org/files/?artifact_id=25355

[gabe-sherman]

We agree that this appears to be an edge case -- however, it is triggering a memory corruption error, which are typically exploitable security vulnerabilities.

Several prior memory corruptions have been fixed by the developers (see #1003, #606), so we feel obligated to bring this one to their attention as well. :)

Sorry for the wall of text, the asan report is below:

==2372851==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000001540 at pc 0x555555622c67 bp 0x7fffffff8dd0 sp 0x7fffffff85a0
READ of size 32 at 0x604000001540 thread T0
    #0 0x555555622c66 in __asan_memcpy (/home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/build/bin/geosop+0xcec66) (BuildId: 88e1b09ac189e14300acabe6ae1a97aceb0037a4)
    #1 0x7ffff779ae9f in geos::geom::CoordinateXYZM geos::algorithm::LineIntersector::intersectionSafe<geos::geom::CoordinateXYZM, geos::geom::Coordinate>(geos::geom::CoordinateXYZM const&, geos::geom::CoordinateXYZM const&, geos::geom::Coordinate const&, geos::geom::Coordinate const&) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/algorithm/LineIntersector.h:604:19
    #2 0x7ffff779a1e7 in geos::geom::CoordinateXYZM geos::algorithm::LineIntersector::intersection<geos::geom::CoordinateXYZM, geos::geom::Coordinate>(geos::geom::CoordinateXYZM const&, geos::geom::CoordinateXYZM const&, geos::geom::Coordinate const&, geos::geom::Coordinate const&) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/algorithm/LineIntersector.h:539:25
    #3 0x7ffff7797314 in unsigned char geos::algorithm::LineIntersector::computeIntersect<geos::geom::CoordinateXYZM, geos::geom::Coordinate>(geos::geom::CoordinateXYZM const&, geos::geom::CoordinateXYZM const&, geos::geom::Coordinate const&, geos::geom::Coordinate const&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/algorithm/LineIntersector.h:459:13
    #4 0x7ffff775922a in void geos::algorithm::LineIntersector::computeIntersection<geos::geom::CoordinateXYZM, geos::geom::Coordinate>(geos::geom::CoordinateXYZM const&, geos::geom::CoordinateXYZM const&, geos::geom::Coordinate const&, geos::geom::Coordinate const&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/algorithm/LineIntersector.h:168:18
    #5 0x7ffff775922a in void geos::algorithm::DoIntersect::operator()<geos::geom::CoordinateXYZM, geos::geom::Coordinate>() /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/algorithm/LineIntersector.cpp:195:14
    #6 0x7ffff775922a in void geos::geom::CoordinateSequences::binaryDispatch<geos::algorithm::DoIntersect>(geos::geom::CoordinateSequence const&, geos::geom::CoordinateSequence const&, geos::algorithm::DoIntersect&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/geom/CoordinateSequences.h:67:86
    #7 0x7ffff775891d in geos::algorithm::LineIntersector::computeIntersection(geos::geom::CoordinateSequence const&, unsigned long, geos::geom::CoordinateSequence const&, unsigned long) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/algorithm/LineIntersector.cpp:212:5
    #8 0x7ffff7aaca8d in geos::noding::IntersectionAdder::processIntersections(geos::noding::SegmentString*, unsigned long, geos::noding::SegmentString*, unsigned long) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/noding/IntersectionAdder.cpp:77:8
    #9 0x7ffff7981eab in geos::index::chain::MonotoneChain::computeOverlaps(geos::index::chain::MonotoneChain const*, double, geos::index::chain::MonotoneChainOverlapAction*) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/index/chain/MonotoneChain.cpp:117:5
    #10 0x7ffff7ab1e23 in geos::noding::MCIndexNoder::intersectChains()::$_0::operator()(geos::index::chain::MonotoneChain const*, geos::index::chain::MonotoneChain const*) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/noding/MCIndexNoder.cpp:72:21
    #11 0x7ffff7ab1e23 in _ZN4geos5index7strtree19TemplateSTRtreeImplIPKNS0_5chain13MonotoneChainENS1_14EnvelopeTraitsEE11visitLeavesIRZNS_6noding12MCIndexNoder15intersectChainsEvE3$_0LDn0EEEbOT_RKNS1_15TemplateSTRNodeIS6_S7_EESJ_ /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/index/strtree/TemplateSTRtree.h:577:16
    #12 0x7ffff7ab1e23 in bool geos::index::strtree::TemplateSTRtreeImpl<geos::index::chain::MonotoneChain const*, geos::index::strtree::EnvelopeTraits>::queryPairs<geos::noding::MCIndexNoder::intersectChains()::$_0&>(geos::index::strtree::TemplateSTRNode<geos::index::chain::MonotoneChain const*, geos::index::strtree::EnvelopeTraits> const&, geos::index::strtree::TemplateSTRNode<geos::index::chain::MonotoneChain const*, geos::index::strtree::EnvelopeTraits> const&, geos::noding::MCIndexNoder::intersectChains()::$_0&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/index/strtree/TemplateSTRtree.h:628:26
    #13 0x7ffff7ab0fac in void geos::index::strtree::TemplateSTRtreeImpl<geos::index::chain::MonotoneChain const*, geos::index::strtree::EnvelopeTraits>::queryPairs<geos::noding::MCIndexNoder::intersectChains()::$_0>(geos::noding::MCIndexNoder::intersectChains()::$_0&&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/index/strtree/TemplateSTRtree.h:305:13
    #14 0x7ffff7ab0fac in geos::noding::MCIndexNoder::intersectChains() /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/noding/MCIndexNoder.cpp:71:11
    #15 0x7ffff7ab0fac in geos::noding::MCIndexNoder::computeNodes(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/noding/MCIndexNoder.cpp:59:5
    #16 0x7ffff7ace1a4 in geos::noding::ValidatingNoder::computeNodes(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/noding/ValidatingNoder.cpp:31:11
    #17 0x7ffff7bc00dd in geos::operation::overlayng::EdgeNodingBuilder::node(std::vector<geos::noding::SegmentString*, std::allocator<geos::noding::SegmentString*> >*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/EdgeNodingBuilder.cpp:104:12
    #18 0x7ffff7bbf7ff in geos::operation::overlayng::EdgeNodingBuilder::build(geos::geom::Geometry const*, geos::geom::Geometry const*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/EdgeNodingBuilder.cpp:88:37
    #19 0x7ffff7bfe0d6 in geos::operation::overlayng::OverlayNG::computeEdgeOverlay() /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNG.cpp:227:46
    #20 0x7ffff7bfb6ef in geos::operation::overlayng::OverlayNG::getResult() /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNG.cpp:182:18
    #21 0x7ffff7bfaab9 in geos::operation::overlayng::OverlayNG::overlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int, geos::geom::PrecisionModel const*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNG.cpp:100:15
    #22 0x7ffff7c058b2 in geos::operation::overlayng::OverlayNGRobust::Overlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNGRobust.cpp:110:18
    #23 0x7ffff78c0c67 in geos::geom::StructuredCollection::doIntersection(geos::geom::StructuredCollection const&) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/geom/HeuristicOverlay.cpp:265:49
    #24 0x7ffff78bb4c1 in geos::geom::HeuristicOverlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/geom/HeuristicOverlay.cpp:66:23
    #25 0x7ffff789546d in geos::geom::Geometry::intersection(geos::geom::Geometry const*) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/geom/Geometry.cpp:540:12
    #26 0x555555782bfc in $_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)::operator()(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeometryOp.cpp:762:33
    #27 0x555555782bfc in Result* std::__invoke_impl<Result*, $_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&>(std::__invoke_other, $_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:61:14
    #28 0x555555782bfc in std::enable_if<__and_<std::__not_<std::is_void<Result*> >, std::is_convertible<std::__invoke_result<$_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&>::type, Result*> >::value, Result*>::type std::__invoke_r<Result*, $_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&>($_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:142:14
    #29 0x555555782bfc in std::_Function_handler<Result* (std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&), $_65::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const::'lambda'(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)>::_M_invoke(std::_Any_data const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:290:9
    #30 0x55555579d01b in std::function<Result* (std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)>::operator()(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&) const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
    #31 0x55555579d01b in GeometryOp::execute(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, double, double) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeometryOp.cpp:1016:35
    #32 0x55555569791a in GeosOp::executeOp(GeometryOp*, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:496:26
    #33 0x5555556960f5 in GeosOp::executeOpRepeat(GeometryOp*, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:480:15
    #34 0x5555556960f5 in GeosOp::executeBinary(GeometryOp*, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:453:30
    #35 0x55555568d64a in GeosOp::execute(GeometryOp*, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:431:9
    #36 0x55555568d64a in GeosOp::run(OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:394:9
    #37 0x55555567e9c6 in main /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:170:12
    #38 0x7ffff6e29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #39 0x7ffff6e29e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #40 0x5555555a0af4 in _start (/home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/build/bin/geosop+0x4caf4) (BuildId: 88e1b09ac189e14300acabe6ae1a97aceb0037a4)

0x604000001540 is located 0 bytes to the right of 48-byte region [0x604000001510,0x604000001540)
allocated by thread T0 here:
    #0 0x55555565e70d in operator new(unsigned long) (/home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/build/bin/geosop+0x10a70d) (BuildId: 88e1b09ac189e14300acabe6ae1a97aceb0037a4)
    #1 0x7ffff7cc9806 in std::__new_allocator<double>::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/new_allocator.h:137:27
    #2 0x7ffff7cc9806 in std::allocator_traits<std::allocator<double> >::allocate(std::allocator<double>&, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/alloc_traits.h:464:20
    #3 0x7ffff7cc9806 in std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:378:20
    #4 0x7ffff7cc9806 in std::vector<double, std::allocator<double> >::reserve(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/vector.tcc:79:22
    #5 0x7ffff7cc9806 in geos::geom::CoordinateSequence::reserve(unsigned long) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/include/geos/geom/CoordinateSequence.h:536:16
    #6 0x7ffff7cc9806 in geos::operation::valid::RepeatedPointRemover::removeRepeatedPoints(geos::geom::CoordinateSequence const*, double) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/valid/RepeatedPointRemover.cpp:84:14
    #7 0x7ffff7bc19a3 in geos::operation::overlayng::EdgeNodingBuilder::removeRepeatedPoints(geos::geom::LineString const*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/EdgeNodingBuilder.cpp:306:12
    #8 0x7ffff7bc19a3 in geos::operation::overlayng::EdgeNodingBuilder::addLine(geos::geom::LineString const*, unsigned char) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/EdgeNodingBuilder.cpp:361:59
    #9 0x7ffff7bbf7c0 in geos::operation::overlayng::EdgeNodingBuilder::build(geos::geom::Geometry const*, geos::geom::Geometry const*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/EdgeNodingBuilder.cpp:87:5
    #10 0x7ffff7bfe0d6 in geos::operation::overlayng::OverlayNG::computeEdgeOverlay() /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNG.cpp:227:46
    #11 0x7ffff7bfb6ef in geos::operation::overlayng::OverlayNG::getResult() /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNG.cpp:182:18
    #12 0x7ffff7bfaab9 in geos::operation::overlayng::OverlayNG::overlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int, geos::geom::PrecisionModel const*) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNG.cpp:100:15
    #13 0x7ffff7c058b2 in geos::operation::overlayng::OverlayNGRobust::Overlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/operation/overlayng/OverlayNGRobust.cpp:110:18
    #14 0x7ffff78c0c67 in geos::geom::StructuredCollection::doIntersection(geos::geom::StructuredCollection const&) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/geom/HeuristicOverlay.cpp:265:49
    #15 0x7ffff78bb4c1 in geos::geom::HeuristicOverlay(geos::geom::Geometry const*, geos::geom::Geometry const*, int) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/geom/HeuristicOverlay.cpp:66:23
    #16 0x7ffff789546d in geos::geom::Geometry::intersection(geos::geom::Geometry const*) const /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/src/geom/Geometry.cpp:540:12
    #17 0x55555579d01b in std::function<Result* (std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&)>::operator()(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&) const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
    #18 0x55555579d01b in GeometryOp::execute(std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, double, double) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeometryOp.cpp:1016:35
    #19 0x55555569791a in GeosOp::executeOp(GeometryOp*, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:496:26
    #20 0x5555556960f5 in GeosOp::executeOpRepeat(GeometryOp*, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, unsigned int, std::unique_ptr<geos::geom::Geometry, std::default_delete<geos::geom::Geometry> > const&, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:480:15
    #21 0x5555556960f5 in GeosOp::executeBinary(GeometryOp*, OpArguments&) /home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/util/geosop/GeosOp.cpp:453:30

SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/gabe/harness_test/fuzzing-libraries/updatedgeos/geos/build/bin/geosop+0xcec66) (BuildId: 88e1b09ac189e14300acabe6ae1a97aceb0037a4) in __asan_memcpy
Shadow bytes around the buggy address:
  0x0c087fff8250: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8260: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8270: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x0c087fff8280: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8290: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
=>0x0c087fff82a0: fa fa 00 00 00 00 00 00[fa]fa fd fd fd fd fd fd
  0x0c087fff82b0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff82c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff82d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff82e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff82f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==2372851==ABORTING

[pramsey]

We'll probably just end up checking the inputs of the function for NaN. It would be nice to filter out NaNs earlier in the process, but that has its own difficulties (WKT and WKB and JSON are not the only ways to put values into the system, see the varies set*() functions, for example) so we end up having to defend against NaN at the top of algorithms instead.

[JamesParrott]

I think I've recreated this, without requiring NaN (or cosmologically-unphysical-extemely-large floating point numbers). This used valgrind, in a Ubuntu Jammy container (hosted on Windows) with geos-bin installed (so Geos 3.10.2 not 3.12.1), and the overlapping squares example from the c_api docs as well as Gabe's.

The detailed error logs, both in Gabe's, and from valgrind in my testing (in the "definitely lost"/error category), both refer to operator new(unsigned long), so perhaps there are just some missing calls to free?

valgrind --leak-check=yes -s geosop -a "POLYGON((0 0, 10 0, 10 10, 0 10, 0 0))" -b "POLYGON((5 5, 1
5 5, 15 15, 5 15, 5 5))" intersection -f txt

produces:

...
==3317== HEAP SUMMARY:
==3317==     in use at exit: 10,036 bytes in 116 blocks
==3317==   total heap usage: 4,160 allocs, 4,044 frees, 196,322 bytes allocated
==3317==
==3317== 161 (144 direct, 17 indirect) bytes in 1 blocks are definitely lost in loss record 66 of 116
==3317==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==3317==    by 0x13E13E: ??? (in /usr/bin/geosop)
==3317==    by 0x117352: ??? (in /usr/bin/geosop)
==3317==    by 0x4CDED8F: (below main) (libc_start_call_main.h:58)
==3317==
...

...
==3317== 257 (144 direct, 113 indirect) bytes in 1 blocks are definitely lost in loss record 116 of 116
==3317==    at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==3317==    by 0x13E13E: ??? (in /usr/bin/geosop)
==3317==    by 0x11859F: ??? (in /usr/bin/geosop)
==3317==    by 0x4CDED8F: (below main) (libc_start_call_main.h:58)
==3317==

==3317== LEAK SUMMARY:
==3317==    definitely lost: 7,344 bytes in 51 blocks
==3317==    indirectly lost: 2,692 bytes in 65 blocks
==3317==      possibly lost: 0 bytes in 0 blocks
==3317==    still reachable: 0 bytes in 0 blocks
==3317==         suppressed: 0 bytes in 0 blocks
==3317==
==3317== ERROR SUMMARY: 51 errors from 51 contexts (suppressed: 0 from 0)

pretty much the same is produced from Gabe's example. There are exactly 51 errors and 116 records in both.

valgrind --leak-check=yes -s geosop -a "GEOMETRYCOLLECTION(POINT(0 0 59.083333333333336), POINT(1 1 59.083333333333336), LINESTRING(2 4 111 1, 2 223 22 2), POLYGON((42 -2.222222223222222e+155 111 1, 2 3 1 1, 2 4 111 NaN, 42 -2.222222223222222e+155 111 NaN)), POLYGON((2.2222222212222224e+149 4 111 1, 42 -2.222222222222222e+149 22 2, 2 4 111 1, 22 2222222222222223 22 NaN, 2.2222222212222224e+149 4 111 NaN)))" -b "LINESTRING (1.6409301755752343e+149 -5.298190649085575e+148, 1.6666666660752404e+149 -5.55555555396982e+148)" intersection -f txt

...

==3316== LEAK SUMMARY:
==3316==    definitely lost: 7,344 bytes in 51 blocks
==3316==    indirectly lost: 2,692 bytes in 65 blocks
==3316==      possibly lost: 0 bytes in 0 blocks
==3316==    still reachable: 0 bytes in 0 blocks
==3316==         suppressed: 0 bytes in 0 blocks
==3316==
==3316== ERROR SUMMARY: 51 errors from 51 contexts (suppressed: 0 from 0)

Incidentally, geosop does compute an answer to Gabe's query (the second line string in "-b"). I suspect it's just an example of garbage in/garbage out, but perhaps someone else is able to interpret it geometrically.

LINESTRING (164093017557523430000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 -52981906490855750000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, 166666666607524040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 -55555555539698200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)

To create the test environment I used:

docker run --rm -it ubuntu:jammy
apt-get update
apt-get install geos-bin
apt-get install valgrind