Skip to content

Commit 444a3a6

Browse files
alambalamb
authored
Donate object_store code from object_store_rs to arrow-rs (apache#2081)
* Import influxdata/object_store_rs@3c51870 * Add object_store to workspace, update notes and readme * Remove old github items * Remove old gitignore * Remove kodiak config * Remove redundant license files * Remove influx specific security policy * Remove redudant rust-toolchain and rustfmt * Add Apache License (RAT) * ignore bubble_up_io_errors test * Fix list_store with explicit lifetime, only run `test_list_root` on linux * Only run object_store throttle tests on a mac
0 parents  commit 444a3a6

File tree

17 files changed

+6261
-0
lines changed

17 files changed

+6261
-0
lines changed

‎.circleci/config.yml‎

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
---
2+
# CI Overview
3+
# -----------
4+
#
5+
# Each night:
6+
#
7+
# A build image is created (ci_image) from `docker/Dockerfile.ci` and is
8+
# pushed to `quay.io/influxdb/rust:ci`. This build image is then used to run
9+
# the CI tasks for the day.
10+
#
11+
# Every commit:
12+
#
13+
# The CI for every PR and merge to main runs tests, fmt, lints and compiles debug binaries
14+
#
15+
# On main if all these checks pass it will then additionally compile in "release" mode and
16+
# publish a docker image to quay.io/influxdb/iox:$COMMIT_SHA
17+
#
18+
# Manual CI Image:
19+
#
20+
# It is possible to manually trigger a rebuild of the image used in CI. To do this, navigate to
21+
# https://app.circleci.com/pipelines/github/influxdata/influxdb_iox?branch=main (overriding the
22+
# branch name if desired). Then:
23+
# - Click "Run Pipeline" in the top-right
24+
# - Expand "Add Parameters"
25+
# - Add a "boolean" parameter called "ci_image" with the value true
26+
# - Click "Run Pipeline"
27+
#
28+
# If you refresh the page you should see a newly running ci_image workflow
29+
#
30+
31+
version: 2.1
32+
33+
orbs:
34+
win: circleci/[email protected]
35+
36+
commands:
37+
rust_components:
38+
description: Verify installed components
39+
steps:
40+
- run:
41+
name: Verify installed components
42+
command: |
43+
rustup --version
44+
rustup show
45+
cargo fmt --version
46+
cargo clippy --version
47+
48+
cache_restore:
49+
description: Restore Cargo Cache
50+
steps:
51+
- restore_cache:
52+
name: Restoring Cargo Cache
53+
keys:
54+
- cargo-cache-{{ arch }}-{{ .Branch }}-{{ checksum "Cargo.lock" }}
55+
- cargo-cache-{{ arch }}-{{ .Branch }}
56+
- cargo-cache
57+
cache_save:
58+
description: Save Cargo Cache
59+
steps:
60+
- save_cache:
61+
name: Save Cargo Cache
62+
paths:
63+
- /usr/local/cargo/registry
64+
key: cargo-cache-{{ arch }}-{{ .Branch }}-{{ checksum "Cargo.lock" }}
65+
66+
jobs:
67+
fmt:
68+
docker:
69+
- image: quay.io/influxdb/rust:ci
70+
environment:
71+
# Disable incremental compilation to avoid overhead. We are not preserving these files anyway.
72+
CARGO_INCREMENTAL: "0"
73+
# Disable full debug symbol generation to speed up CI build
74+
# "1" means line tables only, which is useful for panic tracebacks.
75+
RUSTFLAGS: "-C debuginfo=1"
76+
# https://github.com/rust-lang/cargo/issues/10280
77+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
78+
steps:
79+
- checkout
80+
- rust_components
81+
- cache_restore
82+
- run:
83+
name: Rust fmt
84+
command: cargo fmt --all -- --check
85+
- cache_save
86+
lint:
87+
docker:
88+
- image: quay.io/influxdb/rust:ci
89+
environment:
90+
# Disable incremental compilation to avoid overhead. We are not preserving these files anyway.
91+
CARGO_INCREMENTAL: "0"
92+
# Disable full debug symbol generation to speed up CI build
93+
# "1" means line tables only, which is useful for panic tracebacks.
94+
RUSTFLAGS: "-C debuginfo=1"
95+
# https://github.com/rust-lang/cargo/issues/10280
96+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
97+
steps:
98+
- checkout
99+
- rust_components
100+
- cache_restore
101+
- run:
102+
name: Clippy
103+
command: cargo clippy --all-targets --all-features --workspace -- -D warnings
104+
- cache_save
105+
cargo_audit:
106+
docker:
107+
- image: quay.io/influxdb/rust:ci
108+
environment:
109+
# Disable incremental compilation to avoid overhead. We are not preserving these files anyway.
110+
CARGO_INCREMENTAL: "0"
111+
# Disable full debug symbol generation to speed up CI build
112+
# "1" means line tables only, which is useful for panic tracebacks.
113+
RUSTFLAGS: "-C debuginfo=1"
114+
# https://github.com/rust-lang/cargo/issues/10280
115+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
116+
steps:
117+
- checkout
118+
- rust_components
119+
- cache_restore
120+
- run:
121+
name: Install cargo-deny
122+
command: cargo install --force cargo-deny
123+
- run:
124+
name: cargo-deny Checks
125+
command: cargo deny check -s
126+
- cache_save
127+
check:
128+
docker:
129+
- image: quay.io/influxdb/rust:ci
130+
environment:
131+
# Disable incremental compilation to avoid overhead. We are not preserving these files anyway.
132+
CARGO_INCREMENTAL: "0"
133+
# Disable full debug symbol generation to speed up CI build
134+
# "1" means line tables only, which is useful for panic tracebacks.
135+
RUSTFLAGS: "-C debuginfo=1"
136+
# https://github.com/rust-lang/cargo/issues/10280
137+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
138+
steps:
139+
- checkout
140+
- rust_components
141+
- cache_restore
142+
- run:
143+
name: Install cargo-hack
144+
command: cargo install cargo-hack
145+
- run:
146+
name: Check all features
147+
command: cargo hack check --feature-powerset --no-dev-deps --workspace
148+
- cache_save
149+
doc:
150+
docker:
151+
- image: quay.io/influxdb/rust:ci
152+
environment:
153+
# Disable incremental compilation to avoid overhead. We are not preserving these files anyway.
154+
CARGO_INCREMENTAL: "0"
155+
# Disable full debug symbol generation to speed up CI build
156+
# "1" means line tables only, which is useful for panic tracebacks.
157+
RUSTFLAGS: "-C debuginfo=1"
158+
# https://github.com/rust-lang/cargo/issues/10280
159+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
160+
steps:
161+
- checkout
162+
- rust_components
163+
- cache_restore
164+
- run:
165+
name: Cargo doc
166+
# excluding datafusion because it's effectively a dependency masqueraded as workspace crate.
167+
command: cargo doc --document-private-items --no-deps --workspace --exclude datafusion
168+
- cache_save
169+
- run:
170+
name: Compress Docs
171+
command: tar -cvzf rustdoc.tar.gz target/doc/
172+
- store_artifacts:
173+
path: rustdoc.tar.gz
174+
test:
175+
# setup multiple docker images (see https://circleci.com/docs/2.0/configuration-reference/#docker)
176+
docker:
177+
- image: quay.io/influxdb/rust:ci
178+
- image: localstack/localstack:0.14.4
179+
- image: mcr.microsoft.com/azure-storage/azurite
180+
- image: fsouza/fake-gcs-server
181+
command:
182+
- "-scheme"
183+
- "http"
184+
resource_class: 2xlarge # use of a smaller executor tends crashes on link
185+
environment:
186+
# Disable incremental compilation to avoid overhead. We are not preserving these files anyway.
187+
CARGO_INCREMENTAL: "0"
188+
# Disable full debug symbol generation to speed up CI build
189+
# "1" means line tables only, which is useful for panic tracebacks.
190+
RUSTFLAGS: "-C debuginfo=1"
191+
# https://github.com/rust-lang/cargo/issues/10280
192+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
193+
RUST_BACKTRACE: "1"
194+
# Run integration tests
195+
TEST_INTEGRATION: 1
196+
AWS_DEFAULT_REGION: "us-east-1"
197+
AWS_ACCESS_KEY_ID: test
198+
AWS_SECRET_ACCESS_KEY: test
199+
AWS_ENDPOINT: http://127.0.0.1:4566
200+
AZURE_USE_EMULATOR: "1"
201+
GOOGLE_SERVICE_ACCOUNT: "/tmp/gcs.json"
202+
OBJECT_STORE_BUCKET: test-bucket
203+
steps:
204+
- run:
205+
name: Setup localstack (AWS emulation)
206+
command: |
207+
cd /tmp
208+
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
209+
unzip awscliv2.zip
210+
sudo ./aws/install
211+
aws --endpoint-url=http://localhost:4566 s3 mb s3://test-bucket
212+
- run:
213+
name: Setup Azurite (Azure emulation)
214+
# the magical connection string is from https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=visual-studio#http-connection-strings
215+
command: |
216+
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
217+
az storage container create -n test-bucket --connection-string 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;'
218+
- run:
219+
name: Setup fake GCS server
220+
command: |
221+
curl -X POST --data-binary '{"name":"test-bucket"}' -H "Content-Type: application/json" "http://localhost:4443/storage/v1/b"
222+
echo '{"gcs_base_url": "http://localhost:4443", "disable_oauth": true, "client_email": "", "private_key": ""}' > "$GOOGLE_SERVICE_ACCOUNT"
223+
- checkout
224+
- rust_components
225+
- cache_restore
226+
- run:
227+
name: Cargo test
228+
command: cargo test --workspace --features=aws,azure,azure_test,gcp
229+
- cache_save
230+
231+
test_windows:
232+
executor:
233+
name: win/default
234+
size: medium
235+
environment:
236+
# https://github.com/rust-lang/cargo/issues/10280
237+
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
238+
steps:
239+
- checkout
240+
- run:
241+
name: Download rustup
242+
command: wget https://win.rustup.rs/x86_64 -O rustup-init.exe
243+
- run:
244+
name: Install rustup
245+
command: .\rustup-init.exe -y --default-host=x86_64-pc-windows-msvc
246+
- run:
247+
name: Cargo test
248+
command: cargo test --workspace
249+
250+
workflows:
251+
version: 2
252+
253+
# CI for all pull requests.
254+
ci:
255+
jobs:
256+
- check
257+
- fmt
258+
- lint
259+
- cargo_audit
260+
- test
261+
- test_windows
262+
- doc

‎CONTRIBUTING.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Development instructions
21+
22+
## Running Tests
23+
24+
Tests can be run using `cargo`
25+
26+
```shell
27+
cargo test
28+
```
29+
30+
## Running Integration Tests
31+
32+
By default, integration tests are not run. To run them you will need to set `TEST_INTEGRATION=1` and then provide the
33+
necessary configuration for that object store
34+
35+
### AWS
36+
37+
To test the S3 integration against [localstack](https://localstack.cloud/)
38+
39+
First start up a container running localstack
40+
41+
```
42+
$ podman run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack
43+
```
44+
45+
Setup environment
46+
47+
```
48+
export TEST_INTEGRATION=1
49+
export AWS_DEFAULT_REGION=us-east-1
50+
export AWS_ACCESS_KEY_ID=test
51+
export AWS_SECRET_ACCESS_KEY=test
52+
export AWS_ENDPOINT=http://127.0.0.1:4566
53+
export OBJECT_STORE_BUCKET=test-bucket
54+
```
55+
56+
Create a bucket using the AWS CLI
57+
58+
```
59+
podman run --net=host --env-host amazon/aws-cli --endpoint-url=http://localhost:4566 s3 mb s3://test-bucket
60+
```
61+
62+
Run tests
63+
64+
```
65+
$ cargo test --features aws
66+
```
67+
68+
### Azure
69+
70+
To test the Azure integration
71+
against [azurite](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=visual-studio)
72+
73+
Startup azurite
74+
75+
```
76+
$ podman run -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite
77+
```
78+
79+
Create a bucket
80+
81+
```
82+
$ podman run --net=host mcr.microsoft.com/azure-cli az storage container create -n test-bucket --connection-string 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;'
83+
```
84+
85+
Run tests
86+
87+
```
88+
$ cargo test --features azure
89+
```
90+
91+
### GCP
92+
93+
We don't have a good story yet for testing the GCP integration locally. You will need to create a GCS bucket, a
94+
service account that has access to it, and use this to run the tests.

0 commit comments

Comments
 (0)