Skip to content

feat: distributed Implementation of the Parseable Server #656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 84 commits into from
Feb 24, 2024
Merged

feat: distributed Implementation of the Parseable Server #656

merged 84 commits into from
Feb 24, 2024

Conversation

Eshanatnight
Copy link
Contributor

@Eshanatnight Eshanatnight commented Feb 8, 2024 •
edited
Loading

Description

Refactoring code to implement distributed approach to the parseable server.
This works on a modal basis, by setting P_MODE env variable, or passing the --mode flag with the value.
Default fall back is to original behavior.

@Eshanatnight
Copy link
Contributor Author

Default mode is ready to test.
Deepsource Error is expected.

@nitisht
Copy link
Member

nitisht commented Feb 13, 2024

Default mode is ready to test. Deepsource Error is expected.

Thanks we'll start testing. cc @nikhilsinhacloudsurfex

@Eshanatnight
Copy link
Contributor Author

Fixed a bug in server configure routes

@nitisht nitisht changed the title Distributed Implementation of the Parseable Server feat: distributed Implementation of the Parseable Server Feb 17, 2024
This was referenced Feb 18, 2024
Closed
Closed
@Eshanatnight
wip: broken
c4a450e
@Eshanatnight
ran cargo fmt
392d704
@Eshanatnight
made stuff public on crate level
daa701b 
Might need to change this later
@Eshanatnight
Staring with a Somewhat clean slate
d5977cb
@Eshanatnight
create server.rs + impl ParseableServer Trait
f5d859c
@Eshanatnight
create ssl_acceptor.rs
15b1909
@Eshanatnight
General Implementation
b32c539
@Eshanatnight
chore: move cli handler to a seperate file
d601538
@Eshanatnight
chore: changes to function and cli struct name, to avoid collision
0f2ec9b
@Eshanatnight
fix imports
eb6d427
@Eshanatnight
chore: add LICENSE
d8e28d1
@Eshanatnight
chore: move the constants storage.rs
e40d2d4
@Eshanatnight
chore: replaced sentinals with the constants
0f9c680
@Eshanatnight
fix: forgot default
9c96195
@Eshanatnight
fix: make name more discriptive
abd702a
@Eshanatnight
fix: from_iter not from
d7ca187
@Eshanatnight
using the modal servers
0a50e84
@Eshanatnight
update struct name
e03e639
@Eshanatnight
move server module from modal to http
a2ab7c1
@Eshanatnight
update imports
a56268c
@Eshanatnight
fix changed mode to storage_mode for better clarity
fc84446 
Probably convert to enum so it does not conflict with Server Mode
@Eshanatnight
revert back to module name modal
7852a52
@Eshanatnight
chore: add unit test
8964cf1
@Eshanatnight
fix: ingester file name change
329c727
@Eshanatnight
chore: refactor get_ingestor_info function
a66f7b1
@Eshanatnight
fix: banner server status
fc88c69 
In the banner the LLM Status and the Mode values were switched. Fixed it in this patch
@Eshanatnight
fix: server behaviour when new ingest server is set up
b4d6d23 
When new server is setup, the ingest server staging
should fetch the metadata file from object store
@Eshanatnight
fix: Ingest Server Not Ingesting when attatched to an old data store
04788ce 
When an old data location is used the ingester was not able to ingest.
Due to roles not being set properly.
@Eshanatnight
remove INGESTOR_FILE_EXTENSION constant
691dc7d
@Eshanatnight
chore: remove commented out code
ab17f5f
@Eshanatnight
Trait update
0449fcc
@Eshanatnight
rm rwlock
676c976
@Eshanatnight
chore: clean up
a9676b4
@Eshanatnight
chore: clean up
a333c21
@Eshanatnight
debug out the written buff size
bb7a78a 
for debug only will be removed later
@Eshanatnight
add sync tokio task to sync ingestor info
08ab3a9
@Eshanatnight
fix: bug where query.json was not being created properly
4d4cb30
@Eshanatnight
update: change parquet file names to include port
093fbd0
@Eshanatnight
create a util func to get the sock_addr
311ae8b 
1. Todo: Update with this function whereever needed
@Eshanatnight
manifest file name change
26a1170 
manifest file name is now changed
from            ->  to
manifest.json   ->  ip.port.manifest.json
@Eshanatnight
debug: for working
755994a 
to be uncommented
@Eshanatnight
fix: manifest not being created in multi mode
7f73462
@Eshanatnight
chore: remove debug macros
72186b4
@nikhilsinhaparseable
Copy link
Contributor

@Eshanatnight please fix the checks for fmt and clippy, then we can merge this PR

@nitisht nitisht changed the base branch from main to distributed February 24, 2024 11:19
@nitisht nitisht merged commit 8d3ab29 into parseablehq:distributed Feb 24, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Feb 24, 2024
@Eshanatnight Eshanatnight deleted the distributed branch February 25, 2024 16:54
@Eshanatnight Eshanatnight restored the distributed branch February 25, 2024 16:54
@Eshanatnight Eshanatnight deleted the distributed branch February 25, 2024 16:54
@Eshanatnight Eshanatnight restored the distributed branch February 25, 2024 16:54
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@nitisht nitisht Awaiting requested review from nitisht

@nikhilsinhaparseable nikhilsinhaparseable Awaiting requested review from nikhilsinhaparseable

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Eshanatnight @nitisht @nikhilsinhaparseable