add generic container serialization

[HDembinski]

I noticed that serialization code is missing for boost::container::vector and boost::container::string and probably more of the Boost containers. Boost should be able to serialize ideally all or at least most of its own containers.

The straight forward thing would be to implement serialization code for each individual container, but that's a maintenance burden and it does not exploit the fact that many containers adhere to a standard interface.

With a bit of concept checking a templated generic serialization code for containers could be written that adhere to the vector interface; similar for string, list, and map. The current code for std::vector, std::string, std::map, std::list could be turned into such a generic code. The concept checking would implement a form of duck-typing.

[rwols]

If you're going down this route, I think there should be two forms of "generic" save and load procedures, one for "Sequence" containers and one for "MapLike" containers, e.g. ones that have a key_type and a mapped_type. For some archive implementations this distinction may be beneficial. For instance, YAML has the notion of a sequence and a map.

[robertramey]

On 8/4/18 2:42 AM, Hans Dembinski wrote: I noticed that serialization code is missing for |boost::container::vector| and |boost::container::string| and probably more of the Boost containers. Boost should be able to serialize ideally all or at least most of its own containers. The straight forward thing would be to implement serialization code for each individual container, but that's a maintenance burden and it does not exploit the fact that many containers adhere to a standard interface. With a bit of concept checking a templated generic serialization code for containers could be written that adhere to the vector interface; similar for string, list, and map. The current code for std::vector, std::string, std::map, std::list could be turned into such a generic code. The concept checking would implement a form of duck-typing. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#113>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4R3H2TEhM6Kns-WO4PWz0ZqENbee8nks5uNWx6gaJpZM4Vu7c->.

FYI - the pattern which has been followed from the beginning is: a) if it's a type in the std:: namespace - included serialization (and a test for it) in the serialization library b) otherwise, include serialization in the particular boost library. I would be loath to change this.

…

-- Robert Ramey www.rrsd.com (805)569-3793

[HDembinski]

So, in other words, you think I should open an issue in boost.containers. Fair enough, but my point was to provide generic serialization for standard containers, so that Boost.Serialization would support any vector implementation, be it std, boost, Qt, ...

[alfC]

I agree that serialization of custom classes should be in the respective custom libraries, but "standard stuff" can be part of Boost.Serization, and that includes "concepts".
So, I think it Boost.Serialization could incorporate serialization for "standard concepts"
when they are specific enough to implement serialization in a generic way.

Ultimately, if we (Robert) decide that a concept-based serialization is desirable, I think this will necessarily put the serialization functions in the boost::serialization namespace, otherwise these functions couldn't be found generically (well assuming that all archives types are in
boost::serialization).
This implicitly means that the Boost.Serialization is the obvious place to
put it.

Unfortunately, it looks like there are only a few (official and unofficial)
concepts that are relevant for both serialization and deserialization at this stage of the standard.
(the items are a subset extracted from https://stackoverflow.com/a/24229501/225186)

• TriviallyCopyable

• TrivialType

• StandardLayoutType

• PODType

• Container (of value_type with concept "serializable")

• ReversibleContainer (of "serializable")

• SequenceContainer (of "serializable")

• AssociativeContainer (of "serializable")

• UnorderedAssociativeContainer (of "serializable")

I am not sure how specific they are to actually implement both
serialization and deserialization (and efficiently, for example there is no
concept specific enough to have .reserve(n)).
Without real c++ concepts it could be quite tricky to implement, but it is possible.

I am sure things like this have been partially implemented unintentionally by many people, it is a matter of collecting a bunch of good implementations, for example, in (proposed)

#include<boost/serialization/std/concepts/Container.hpp> // e.g. to serialize (any container) only, no specific enough to deserialize
#include<boost/serialization/std/concepts/SequenceContainer.hpp> // e.g. to deserialize (no much gain in serialize) (things like std::vector), based on `push_back` (`reserve` could be detected?)
#include<boost/serialization/std/concepts/AssociativeContainer.hpp> // e.g. to deserialize (things like std::map or std::set?), based on `insert`.
#include<boost/serialization/std/concepts/UnorderredAssociativeContainer // e.g. to deserialize (things like std::unordered_map/set)

As (standard) concepts become more specific or general more implementations
(and more optimal even) can be added.

I think this setups a more formal way to discuss this issue.

[robertramey]

This is an interesting idea. It looks like you've peered into the implementation of std container serialization and seen how i've attempted to factor out the commonality in the implementations. So the idea making this more general using concepts is an interesting one and very much in line with the design and subsequence maintenance of the serialization library. I believe that his approach is probably the main reason it has managed to stay relevant after so many years. So I think the idea is quite interesting but seems to me it would require a lot more investigation:

a. concepts isn't yet in C++ and people are still arguing about. I've approached serialization as "solving the problem we have" rather than "this is the next great thing". I'm not sure you have enough to work with yet.

b. it's very unclear to me how the "overloading" feature of concepts would play in to this. I might or might not be useful. interesting though.

c. If one were to make a module which implemented serialization for any components which fullfill std concepts. I agree that the place to place would be in the serialization library.

d. the one's you mention above, sequence container, etc. are already factored out and implemented within the serialization library implementation. These are done in terms of specific containers as type parameters. So much of the implementation is "done"

e. this would entail a major, major, major effort in terms of documentation, tutorial, examples, etc. If I were to undertake this, I'd recommend that the serialization transition to boost book or doc book first.
Same can said for testing.

f. The library does have a good foundation of concepts in the documentation and there is (ad hock) compile time testing of concept definitions. But this is ad hoc and would likely have to be re-visited to make it consistent with this effort.

g. Then there is the question of backward compatibility of already created archives. Always an issue. In this case, it would be stickier than other cases.

In short, an interesting idea worthy of exploration. But I can't really know where it would lead.

[correaa]

That's right Robert. Programming with concepts requires us to think differently (and think really hard). The old paradigm, based on overloads, specific types and matching templates arguments, and the new paradigm can coexists.

I think the weak point is that many concepts still have a long way to go to be specific enough to implement serializaliation (and specially deserialization since "mutability" concepts are less intuitive), let alone efficiently. But slowly getting there wouldn't harm and will help identify the concepts we need. Eventually, concept-based serialization will be equivalent and as efficient to already implemented cases (e.g. std::vector, std::set, std::map) but will be "open" to new containers. This will minimize code and code coupling to an optimal minimum.

The hard part is to identify concepts and hope that the standard introduce good basic concepts to build upon. This is a very interesting project.

BTW, in SE, there is a intriguing posting by Sean Parent https://stackoverflow.com/a/14000046/225186 where he defined the theoretical 'area' function (and "underlying type") for Regular types. In some sense the 'area' is the "size of serialization" (or an upper bound?) and is something to think in relation to concept-based serialization. (this is a generalization to the size() function of a container).

[robertramey]

On 8/5/18 1:52 PM, Alfredo Correa wrote: That's right Robert. Programming with concepts requires us to think differently (and think really hard). The old paradigm, based on overloads, specific types and matching templates arguments, and the new paradigm can coexists.

I don't think Its not that much different by the time one gets it implemented.

I think the weak point is that many concepts still have a long way to go

Right ...

 slowly getting there wouldn't harm and will help identify the concepts we need. Eventually, concept-based serialization will be equivalent and as efficient to already implemented cases (e.g. std::vector, std::set, std::map) but will be "open" to new containers. This will minimize code and code coupling to an optimal minimum. The hard part is to identify concepts and hope that the standard introduce good basic concepts to build upon. This is a very interesting project.

Actually I see a way to work on this without mucking anything else up. Implement serialization for boost/collections in terms of std::concepts.  The actual implemention code can be stolen from the current implementation of std::vector, std::map, etc.

BTW, in SE, there is a intriguing posting by Sean Parent https://stackoverflow.com/a/14000046/225186 where he defined the theoretical 'area' function (and "underlying type") for Regular types. In some sense the 'area' is the "size of serialization" (or an upper bound?) and is something to think in relation to concept-based serialization. (this is a generalization to the |size()| function of a container).

I don't find this convincing or anyway helpful. If you want a nice project implement serialization for boost collections.  But do it in way such that applies to any type with specificly define type requirements (concepts).  Then even if it doesn't work - you'll still have added serialization into an type which might benefit from it.

…

-- Robert Ramey www.rrsd.com (805)569-3793

[correaa]

On Sun, Aug 5, 2018 at 2:05 PM Robert Ramey <notifications@github.com> wrote:

On 8/5/18 1:52 PM, Alfredo Correa wrote: > Programming with concepts requires us to think > differently (and think really hard). The old paradigm, based on > overloads, specific types and matching templates arguments, and the > new paradigm can coexists. > I don't think Its not that much different by the time one gets it implemented.

Sure, actual code will look like everyday code in the end. It is just that, given certain requirements one has to think hard exactly how to implement the optimal or acceptable (de)serialization. Think, for example, of whether to use `reserve` or not given a concept.

> > The hard part is to identify concepts and hope that the standard > introduce good basic concepts to build upon. This is a very > interesting project. > Actually I see a way to work on this without mucking anything else up.

Great, are you thinking in the lines of std::enable_if, general decltype/detect SFINAE or C++ Concepts-lite?

Implement serialization for boost/collections in terms of std::concepts. The actual implemention code can be stolen from the current implementation of std::vector, std::map, etc.

I see, yes, that is called "lifting" the code to get more generic code. The challenge is to lift the code maintaining an acceptable efficiency (again, think of the example of `reserve`).

> > BTW, in SE, there is a intriguing posting by Sean Parent > https://stackoverflow.com/a/14000046/225186 where he defined the > theoretical 'area' function (and "underlying type") for Regular types. > In some sense the 'area' is the "size of serialization" (or an upper > bound?) and is something to think in relation to concept-based > serialization. (this is a generalization to the |size()| function of a > container). > I don't find this convincing or anyway helpful.

Yes, there is not a urgent need for this yet. But, I think the idea of `underlying_type` and `area` are deeply related to bit-based representations of regular types (aka serialization). That is why I brought the topic, so we start to think in terms of formal concepts. (My other point is that any collection serialization that uses `size()` is a particular case of the more general idea of `area`). Also, I can imagine archive types that can benefit from knowing in advance upper bounds to the size of its elements.

If you want a nice project implement serialization for boost collections. But do it in way such that applies to any type with specificly define type requirements (concepts). Then even if it doesn't work - you'll still have added serialization into an type which might benefit from it.

Yes, I have some code like this, that is why I have strong opinions. Having said all that, I would say the hardest part is how informally defined and how badly scattered is the documentation of concepts in the C++ standard. I think the first step is to identify the concepts we would like to use and implement, that is why I put this initial list. We have to understand that the correspondence between serialization and deserialization will be broken for some concepts. For example, a Sequence can be always serialized, but it can be deserialized only when the target size is the expected one (because no concept that I know of actually has the requirement of a resize). That is why I say that deserialization based on concepts probably is still challenging without further refinements (for example ResizableSequence or ElasticSequence, it doesn't have a name as far as I know).

[HDembinski]

I don't see why it is necessary to push the discussion towards concepts. We have SFINAE and enable_if, even in pre C++11. With this you can make a generic serialization function for vector-like containers that have methods required for the serialization to work, namely reserve(n) etc. There is no need to make this more complicated.

[HDembinski]

On top of the generic version, there can be specialized overloads for specific vector-types if needed. The compiler will use the generic version only as a last resort. The same approach can be used for set-like, map-like etc., I just want to keep the discussion simple and concrete for now, so I focus on vector-like.

[sdebionne]

I would also be interested in generic serialization.

To use SFINAE, the default implementation of serialize(), would need to be "removed" else one would get ambiguous overload issues.

For instance, with:

template <typename Archive, typename T>
void serialize(Archive& ar, T& d, const unsigned int version, std::enable_if_t< is_bus_stop<T>::value >* = nullptr) {}

I get:

error: call of overloaded 'serialize(boost::archive::text_oarchive&, bus_stop&, const boost::serialization::version_type&)' is ambiguous
     serialize(ar, t, v);
     ~~~~~~~~~^~~~~~~~~~
boost/serialization/serialization.hpp:65:13: note: candidate: void boost::serialization::serialize(Archive&, T&, unsigned int) [with Archive = boost::archive::text_oarchive; T = person]
 inline void serialize(
             ^~~~~~~~~
other/serialization.hpp:24:8: note: candidate: void boost::serialization::serialize(Archive&, T&, unsigned int, std::enable_if_t<is_bus_stop<T>::value>*) [with Archive = boost::archive::text_oarchive; T = person; std::enable_if_t<boost::hana::Struct<T>::value> = void]
   void serialize(Archive& ar, T& d, const unsigned int version, std::enable_if_t<hana::Struct<T>::value>* = nullptr)
        ^~~~~~~~~

Looking at the boost/serialization/serialization.hpp header, it looks like a workaround has already been implemented, but for a different purpose (compilers lacking proper PTFO support) :

// note the usage of function overloading to compensate that C++ does not
// currently support Partial Template Specialization for function templates 
// We have declared the version number as "const unsigned long".  
// Overriding templates for specific data types should declare the version
// number as "const unsigned int". Template matching will first be applied
// to functions with the same version types - that is the overloads.  
// If there is no declared function prototype that matches, the second argument
// will be converted to "const unsigned long" and a match will be made with 
// one of the default template functions below.

The workaround was removed in 5a94e5e though.

[sdebionne]

We could add some level of reflection to check if a given class has a serialize member function, something along the (c++11) lines of:

#include <type_traits>

template <typename Archive, typename C>
struct has_serialize
{
private:
    template<typename T>
    static constexpr auto check(T*)
        -> typename
        std::is_same<
            decltype(std::declval<T>().serialize(std::declval<Archive&>(), std::declval<unsigned int>())),
            void
        >::type;

    template<typename>
    static constexpr std::false_type check(...);

    typedef decltype(check<C>(0)) type;

public:
    static constexpr bool value = type::value;
};

The default implementation would then be easily disabled with:

// default implementation - call the member function "serialize"
template <class Archive, class T>
inline void serialize(
	Archive & ar, T & t, const unsigned int file_version, std::enable_if_t<has_serialize<Archive, T>::value>* = nullptr
){
	access::serialize(ar, t, static_cast<unsigned int>(file_version));
}

@HDembinski Is that a path worth exploring?

[HDembinski]

@sdebionne I think we have different things in mind. My point was the following: to serialize a vector-like containter, you don't need to access its internals. Any container that supports the vector interface can be (de)serialized with the same templated code, something along the lines of (pseudo-code):

template <class Archive, class Vector>
void save(Archive& ar, const Vector& v, unsigned version) {
   ar << v.size();
   ar << boost::serialization::make_array(v.data(), v.size());
}

template <class Archive, class Vector>
void load(Archive& ar, Vector& v, unsigned version) {
   std::size_t size = 0;
   ar >> size;
   v.reset(size);
   ar >> boost::serialization::make_array(v.data(), v.size());
}

template<class Archive, class Vector>
inline void serialize(
    Archive & ar,
    Vector & t,
    const unsigned int file_version
){
    boost::serialization::split_free(ar, t, file_version);
}

To avoid ambiguities, these three functions may only participate in overload resolution when the class Vector implements the vector interface, in particular it must have a .size() method, a .data() method and a .reset() method. This can be checked with SFINAE tricks even in pre-C++11.

[HDembinski]

@sdebionne Your decltype code is C++11, so it cannot be used in Boost.Serialization. I am sure Robert doesn't want to drop support for pre-C++11 compilers. Boost.Serialization code could be much simpler if support for old language standards was dropped.

[sdebionne]

@HDembinski My point was that a generic version for Vector-like containers will be ambiguous with the default implementation that comes with Boost.Serialization

// default implementation - call the member function "serialize"
template <class Archive, class T>
inline void serialize(
	Archive & ar, T & t, const unsigned int file_version) {
	access::serialize(ar, t, static_cast<unsigned int>(file_version));
}

that needs to be disabled somehow (or have lower priority). Or am I missing something?

struct has_serialize is implemented in C++11 for an easy solution, but it probably does not have to.