Suggestion: Initial sync

[calexandre]

Hey Richard,
I would like to suggest some sort of initial sync functionality (optional).

Something like when you create the river via the PUT api, some additional options regarding on how the user would like to perform the initial sync.

This would be a "one time" operation. I dont even know if it is possible...

The main issue is that not everything is on the oplog, especially for really large and stale collections...
So it would be nice to implement a set of options that would allow the user to tell the river to pull all data from mongo (much like a GetAll operation).

Of course we could discuss different strategies for pulling the data, such as:

1. GetAll (easy, but cumbersome for large collections)
2. via MongoDump, MongoExport of BsonDump
3. Others..?

It would be nice to support different import strategies, much like as plugins for this river.

Keep up the good work :)

[xma]

Hello,

I'd agree if that could be exposed by an API or any other way (conf. file / etc ...) that would be really great.

For stale collection (at least my use case I would say) I've made a patch
#45

and I don't like idea of using personal forked code suited for my use case, in production.

Regards,

[greatwitenorth]

Yes I'd also love to see this feature implemented. After working with the mysql river (which slurps in the table initially) I thought I was doing something wrong when my collection wasn't being slurped. If there is no plan to implement this, it might be worth mentioning this in the wiki.

[medcl]

+1 once we have changed the mapping,we need to clean the old-index,and ask a "re-pull" function,pulling data from mongo-db to elasticsearch,hope to see this feature.

[mzafer]

+1. Would love to see this feature supported

[subratbasnet]

+1 would love this too!

[bitmorse]

+1 for this!

[subratbasnet]

A good work around for this would be to simply do a BULK-UPDATE on the collection after the mongo-rivers are setup. I use this for millions of records and it works great

[enrique-fernandez-polo]

+1 Very usefull!

[yvesx]

+1

[yvesx]

subratbasnet:
what do yo u mean by a BULK-UPDATE? if A is the large stale collection, do you mean setting up another empty collection B do this:
db.collectionA.find().
forEach( function(i) {
i.ts_imported = new Date();
db.collectionB.insert(i);
});

Then setup the river on collectionB?

[subratbasnet]

Yevesx:

What I meant was, when you have a stale collection in mongo. First you would setup the river for that collection. This will NOT automatically start moving the data from the stale collection to Elastic search.

To trigger that, you could simply perform a bulk update on the collection by a specific condition that matches all the records. For example, in my case, I simply change the "updated" field inside all the documents in my collection.. and this triggers the river and it moves the affected documents to Elaseicsearch

[yvesx]

I see! This is a clever idea.
On May 17, 2013, at 10:59 AM, Subrat Basnet notifications@github.com wrote:

Yevesx:

What I meant was, when you have a stale collection in mongo. First you would setup the river for that collection. This will NOT automatically start moving the data from the stale collection to Elastic search.

To trigger that, you could simply perform a bulk update on the collection by a specific condition that matches all the records. For example, in my case, I simply change the "updated" field inside all the documents in my collection.. and this triggers the river and it moves the affected documents to Elaseicsearch

—
Reply to this email directly or view it on GitHub.