DOCS-11102: clarify sort-limit coalescence behavior

Author: schmalliso

source/core/aggregation-pipeline-optimization.txt

@@ -169,72 +169,32 @@ The optimizer can add the same :pipeline:`$match` stage before the
    { $redact: { $cond: { if: { $eq: [ "$level", 5 ] }, then: "$$PRUNE", else: "$$DESCEND" } } },
    { $match: { year: 2014, category: { $ne: "Z" } } }
 
-.. _agg-skip-limit-optimization:
+.. _agg-project-skip-optimization:
 
-``$skip`` + ``$limit`` Sequence Optimization
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When you have a sequence with :pipeline:`$skip` followed by a
-:pipeline:`$limit`, the :pipeline:`$limit` moves before the
-:pipeline:`$skip`. With the reordering, the :pipeline:`$limit` value
-increases by the :pipeline:`$skip` amount.
-
-For example, if the pipeline consists of the following stages:
-
-.. code-block:: javascript
-
-   { $skip: 10 },
-   { $limit: 5 }
-
-During the optimization phase, the optimizer transforms the sequence to
-the following:
-
-.. code-block:: javascript
-
-   { $limit: 15 },
-   { $skip: 10 }
-
-This optimization allows for more opportunities for
-:ref:`agg-sort-limit-coalescence`, such as with ``$sort`` + ``$skip`` +
-``$limit`` sequences. See :ref:`agg-sort-limit-coalescence` for details
-on the coalescence and :ref:`agg-sort-skip-limit-sequence` for an
-example.
-
-For aggregation operations on :doc:`sharded collections
-<aggregation-pipeline-sharded-collections>`, this optimization reduces
-the results returned from each shard.
-
-.. _agg-project-skip-limit-optimization:
-
-``$project`` + ``$skip`` or ``$limit`` Sequence Optimization
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``$project`` + ``$skip`` Sequence Optimization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. versionadded:: 3.2
 
-When you have a sequence with :pipeline:`$project` followed by either
-:pipeline:`$skip` or :pipeline:`$limit`, the :pipeline:`$skip` or
-:pipeline:`$limit` moves before :pipeline:`$project`. For example, if
+When you have a sequence with :pipeline:`$project` followed by 
+:pipeline:`$skip`, the :pipeline:`$skip` 
+moves before :pipeline:`$project`. For example, if
 the pipeline consists of the following stages:
 
 .. code-block:: javascript
 
    { $sort: { age : -1 } },
    { $project: { status: 1, name: 1 } },
-   { $limit: 5 }
+   { $skip: 5 }
 
 During the optimization phase, the optimizer transforms the sequence to
 the following:
 
 .. code-block:: javascript
 
    { $sort: { age : -1 } },
-   { $limit: 5 }
-   { $project: { status: 1, name: 1 } },
-
-This optimization allows for more opportunities for
-:ref:`agg-sort-limit-coalescence`, such as with ``$sort`` + ``$limit``
-sequences. See :ref:`agg-sort-limit-coalescence` for details on the
-coalescence.
+   { $skip: 5 },
+   { $project: { status: 1, name: 1 } }
 
 .. _aggregation-pipeline-coalescence-optimization:
 
@@ -250,14 +210,58 @@ reordering optimization.
 ``$sort`` + ``$limit`` Coalescence
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-When a :pipeline:`$sort` immediately precedes a :pipeline:`$limit`, the
-optimizer can coalesce the :pipeline:`$limit` into the
-:pipeline:`$sort`. This allows the sort operation to only maintain the
+.. versionchanged:: 4.0
+
+When a :pipeline:`$sort` precedes a :pipeline:`$limit`, the optimizer
+can coalesce the :pipeline:`$limit` into the :pipeline:`$sort` if no
+intervening stages modify the number of documents 
+(e.g. :pipeline:`$unwind`, :pipeline:`$group`).
+MongoDB will not coalesce the :pipeline:`$limit` into the
+:pipeline:`$sort` if there are pipeline stages that change the number of
+documents between the :pipeline:`$sort` and :pipeline:`$limit` stages..
+
+For example, if the pipeline consists of the following stages:
+
+.. code-block:: javascript
+
+   { $sort : { age : -1 } },
+   { $project : { age : 1, status : 1, name : 1 } },
+   { $limit: 5 }
+
+During the optimization phase, the optimizer coalesces the sequence
+to the following:
+
+.. code-block:: javascript
+
+   {
+       "$sort" : {
+          "sortKey" : {
+             "age" : -1
+          },
+          "limit" : NumberLong(5)
+       }
+   },
+   { "$project" : { 
+            "age" : 1, 
+            "status" : 1, 
+            "name" : 1 
+     } 
+   }
+
+This allows the sort operation to only maintain the
 top ``n`` results as it progresses, where ``n`` is the specified limit,
 and MongoDB only needs to store ``n`` items in memory
 [#coalescence-allowDiskUse]_. See :ref:`sort-and-memory` for more
 information.
 
+.. admonition:: Sequence Optimization with $skip
+
+   If there is a :pipeline:`$skip` stage between the :pipeline:`$sort`
+   and :pipeline:`$limit` stages, MongoDB will coalesce the
+   :pipeline:`$limit` into the :pipeline:`$sort` stage and increase the
+   :pipeline:`$limit` value by the :pipeline:`$skip` amount. See
+   :ref:`agg-sort-skip-limit-sequence` for an example.
+
 .. [#coalescence-allowDiskUse] The optimization will still apply when
    ``allowDiskUse`` is ``true`` and the ``n`` items exceed the
    :ref:`aggregation memory limit <agg-memory-restrictions>`.
@@ -378,13 +382,8 @@ option, the ``explain`` output shows the coalesced stage:
      }
    }
 
-Examples
---------
-
-The following examples are some sequences that can take advantage of
-both sequence reordering and coalescence. Generally, coalescence occurs
-*after* any sequence reordering optimization.
-
+Example
+-------
 .. _agg-sort-skip-limit-sequence:
 
 ``$sort`` + ``$skip`` + ``$limit`` Sequence
@@ -399,60 +398,24 @@ A pipeline contains a sequence of :pipeline:`$sort` followed by a
    { $skip: 10 },
    { $limit: 5 }
 
-First, the optimizer performs the :ref:`agg-skip-limit-optimization` to
+The optimizer performs :ref:`agg-sort-limit-coalescence` to
 transforms the sequence to the following:
 
 .. code-block:: javascript
 
-   { $sort: { age : -1 } },
-   { $limit: 15 }
-   { $skip: 10 }
-
-The :ref:`agg-skip-limit-optimization` increases the :pipeline:`$limit`
-amount with the reordering. See :ref:`agg-skip-limit-optimization` for
-details.
-
-The reordered sequence now has :pipeline:`$sort` immediately preceding
-the :pipeline:`$limit`, and the pipeline can coalesce the two stages to
-decrease memory usage during the sort operation. See
-:ref:`agg-sort-limit-coalescence` for more information.
-
-``$limit`` + ``$skip`` + ``$limit`` + ``$skip`` Sequence
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-A pipeline contains a sequence of alternating :pipeline:`$limit` and
-:pipeline:`$skip` stages:
-
-.. code-block:: javascript
-
-   { $limit: 100 },
-   { $skip: 5 },
-   { $limit: 10 },
-   { $skip: 2 }
-
-The :ref:`agg-skip-limit-optimization` reverses the position of the ``{
-$skip: 5 }`` and ``{ $limit: 10 }`` stages and increases the limit
-amount:
-
-.. code-block:: javascript
-
-   { $limit: 100 },
-   { $limit: 15},
-   { $skip: 5 },
-   { $skip: 2 }
-
-The optimizer then coalesces the two :pipeline:`$limit` stages into a
-single :pipeline:`$limit` stage and the two :pipeline:`$skip` stages
-into a single :pipeline:`$skip` stage. The resulting sequence is the
-following:
-
-.. code-block:: javascript
-
-   { $limit: 15 },
-   { $skip: 7 }
+   {
+      "$sort" : {
+         "sortKey" : {
+            "age" : -1
+         },
+         "limit" : NumberLong(15)
+      }
+   },
+   {
+      "$skip" : NumberLong(10)
+   }
 
-See :ref:`agg-limit-limit-coalescence` and
-:ref:`agg-skip-skip-coalescence` for details.
+MongoDB increases the :pipeline:`$limit` amount with the reordering.
 
 .. seealso:: 
    :method:`explain <db.collection.aggregate()>` option in the

source/includes/fact-agg-sort-limit.rst

@@ -1,7 +1,9 @@
-When a :pipeline:`$sort` immediately precedes a :pipeline:`$limit` in
-the pipeline, the :pipeline:`$sort` operation only maintains the top
-``n`` results as it progresses, where ``n`` is the specified limit, and
-MongoDB only needs to store ``n`` items in memory. This optimization
-still applies when ``allowDiskUse`` is ``true`` and the ``n`` items
-exceed the :ref:`aggregation memory limit <agg-memory-restrictions>`.
-
+When a :pipeline:`$sort` precedes a :pipeline:`$limit` and there are no
+intervening stages that modify the number of documents, the optimizer can
+coalesce the :pipeline:`$limit` into the :pipeline:`$sort`. This allows
+the :pipeline:`$sort` operation to only
+maintain the top ``n`` results as it progresses, where ``n`` is the
+specified limit, and ensures that MongoDB only needs to store ``n`` items in memory.
+This optimization still applies when ``allowDiskUse`` is ``true`` and
+the ``n`` items exceed the :ref:`aggregation memory limit
+<agg-memory-restrictions>`.