improve observability a bit, simplify sink

[arthurpassos]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

1. Properly implement system.exports table
2. Add profileevents
3. Remove importer sink, simplified the progress callback
4. Add setting to overwrite file if it already exists in remote storage
5. Store both parallel formatting and setting in the manifest
6. Add part checksum to the filename

Documentation entry for user-facing changes

...

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

[github-actions]

Workflow [PR], commit [437e67c]

[mkmkme]

I'm not sure what is going on here... Are you trying to do the same thing in case writePartLog throws? Or is it only exec.execute() that can throw in this context?

[arthurpassos]

It should only be exec.execute() that throws. And we need to write the log in both cases

[filimonov]

maybe smth like:

Suggested change

	tryLogCurrentException(__PRETTY_FUNCTION__, "Exception is in export part task");
	tryLogCurrentException(__PRETTY_FUNCTION__, fmt::format("while exporting the part {}. User should retry.", manifest.data_part->name));

[filimonov]

i would probably add few counters to profile events smth like
PartsExports (counter of successfull exports)
PartsExportFailures (counter of exports with exception)
PartsExportDuplicated (when target exists)
PartsExportBytes (sum of the bytes written by exports)
PartsExportTotalMilliseconds (overall time of exports)

(Names - if you will have better ideas - feel free :) )

[arthurpassos]

Added these except for PartsExportBytes because I still don't have the writing stats integrated.

[filimonov]

If you can populate ProfileEvents -it would be awesome. (it also strored in thread context, probably not too hard to add but not 100% sure, but it's not a blocker, can go w/o)

Additionally i would like to see in system.exports the target file name. Ideally in the part_log too (maybe reuse path_on_disk column?)

[arthurpassos]

Also added the setting to overwrite the file

[Enmk]

Implementation overview

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    ExportsList *--> "*" ExportsListElement
    ExportsListElement *--> ThreadGroup
    ContextShared *--> ExportsList
    Context --> ExportsList : exposes

    ExportsListElement --> ExportInfo : creates
    StorageSystemExports --> ExportInfo : reads

    MergeTreeData "*" *--> MergeTreeExportManifest
    MergeTreeData --> ExportsList : adds elemets

Loading

[arthurpassos]

Implementation overview

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
    ExportsListElement --> ExportInfo : creates

    ExportsList *--> "*" ExportsListElement
    ContextShared *--> ExportsList
    Context --> ExportsList : exposes

    StorageSystemExports --> ExportInfo : reads
    MergeTreeData --> MergeTreeExportManifest
    MergeTreeData --> ExportsList : adds elemets

Loading

That seems about right

[Enmk]

LGTM