diff --git a/docs/dir/.index b/docs/dir/.index index 458c137..cebacae 100644 --- a/docs/dir/.index +++ b/docs/dir/.index @@ -6,6 +6,9 @@ nav: - Getting Started: getting-started.md - Features and Usage: scenarios.md - Event Streaming: events.md + - Directory Sandbox Environment: + - Public Staging Instance: directory-public-staging.md + - Deployment: directory-sandbox-deployment.md - Hosted AGNTCY Agent Directory: hosted-agent-directory.md - CLI Reference: directory-cli.md - SDK Reference: directory-sdk.md diff --git a/docs/dir/directory-public-staging.md b/docs/dir/directory-public-staging.md new file mode 100644 index 0000000..bbc492a --- /dev/null +++ b/docs/dir/directory-public-staging.md @@ -0,0 +1,295 @@ +# Public Staging Instance + +Welcome to the **Directory Public Staging Environment** - a place to develop and test +with the decentralized AI agent discovery network. +This environment provides a fully functional Directory instance for development, testing, and exploration purposes. + +This is a public staging environment for development and testing. Keep in mind the following: + +* There are no SLA or data persistence guarantees. +* This environment is not for production use. +* This environment is ideal for prototyping, integration, and exploration. + +## Architecture Overview + +``` +┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐ +│ Your Application │ │ Directory Network │ │ Other Federation │ +│ │ │ │ │ Members │ +│ ┌─────────────┐ │ │ ┌──────────────────┐ │ │ │ +│ │ Directory │◄───┼────┼►│ Directory API │ │ │ ┌─────────────────┐ │ +│ │ Client SDK │ │ │ │ Service │ │ │ │ Partner Org │ │ +│ └─────────────┘ │ │ └──────────────────┘ │ │ │ Directory │ │ +│ │ │ │ │ │ Instances │ │ +│ ┌─────────────┐ │ │ ┌──────────────────┐ │◄───┼─┤ │ │ +│ │ SPIRE Agent │◄───┼────┼►│ SPIRE Server │ │ │ └─────────────────┘ │ +│ └─────────────┘ │ │ │ (Federation) │ │ │ │ +└─────────────────────┘ │ └──────────────────┘ │ └─────────────────────┘ + └──────────────────────┘ +``` + +## Available Endpoints + +| Service | URL | Purpose | +| -------------------- | ------------------------------------- | ------------------------------------------- | +| **Directory API** | `https://api.directory.agntcy.org` | Main API for agent discovery and management | +| **SPIRE Federation** | `https://spire.directory.agntcy.org` | SPIRE server for secure identity federation | +| **Status Dashboard** | `https://status.directory.agntcy.org` | Real-time service status and monitoring | + +## Quick Start Guide + +### Prerequisites + +Before you begin, ensure you have: + +- A SPIRE server setup in your organization. +- Basic understanding of SPIFFE/SPIRE concepts. +- Directory client SDK or CLI tools available. + +### Prepare Your Environment + +#### Option 1: Using Directory CLI + +1. **Install the CLI**: + ```bash + # Using Homebrew (Linux/macOS) + brew tap agntcy/dir https://github.com/agntcy/dir + brew install dirctl + + # Or download directly from releases + curl -L https://github.com/agntcy/dir/releases/latest/download/dirctl-linux-amd64 -o dirctl + chmod +x dirctl + sudo mv dirctl /usr/local/bin/ + ``` + +2. **Configure the client**: + ```bash + dirctl config set server-address api.directory.agntcy.org + dirctl config set spiffe-socket-path /tmp/spire-agent/public.sock + ``` + +3. **Test the connection**: + ```bash + dirctl ping + # Expected: ✅ Connected to Directory API at api.directory.agntcy.org + ``` + +#### Option 2: Using Directory Client SDK + +Choose your preferred language: + +
+Go SDK + +```go +package main + +import ( + "context" + "log" + + "github.com/agntcy/dir/client" +) + +func main() { + // Create client with SPIRE support + config := &client.Config{ + ServerAddress: "api.directory.agntcy.org", + SpiffeSocketPath: "/tmp/spire-agent/public.sock", + } + client, _ := client.New(client.WithConfig(config)) + + // Test connection + _, err := client.Ping(context.Background()) + if err != nil { + log.Printf("❌ Connection failed: %v", err) + return + } + + log.Println("✅ Connected to Directory!") + + // Run workflows... +} +``` +
+ +
+Python SDK + +```python +from agntcy.dir_sdk.client import Config, Client + +def main(): + # Create client with SPIRE support + config = Config( + server_address="api.directory.agntcy.org", + spiffe_socket_path="/tmp/spire-agent/public.sock" + ) + client = Client(config) + + # Test connection + try: + client.ping() + print("✅ Connected to Directory!") + except Exception as e: + print(f"❌ Connection failed: {e}") + + # Run workflows... + +if __name__ == "__main__": + main() +``` +
+ +
+JavaScript SDK + +```javascript +import {Config, Client} from 'agntcy-dir'; + +async function main() { + // Create client with SPIRE support + const config = new Config({ + serverAddress: "api.directory.agntcy.org", + spiffeEndpointSocket: "/tmp/spire-agent/public.sock", + }); + const transport = await Client.createGRPCTransport(config); + const client = new Client(config, transport); + + // Test connection + try { + await client.ping(); + console.log('✅ Connected to Directory!'); + } catch (error) { + console.error('❌ Connection failed:', error.message); + } + + // Run workflows... +} + +main(); +``` + +Note that this SDK is intended for Node.js applications only and does not work in web browsers. + +
+ +### Federation Setup (Required) + +To interact with the Directory, you need to establish a trusted federation between your SPIRE server and the Directory SPIRE server. + +### Step 1: Prepare Your Federation Request + +Create a file with your SPIRE server details using the template below: + +```yaml +# onboarding/your-org.com.yaml +trustDomain: your-org.com +bundleEndpointURL: https://spire.your-org.com +bundleEndpointProfile: + type: https_spiffe + endpointSPIFFEID: spiffe://your-org.com/spire/server +trustDomainBundle: |- + { + "keys": [ + { + "use": "x509-svid", + "kty": "RSA", + "n": "your-public-key-here...", + "e": "AQAB", + "x5c": ["your-certificate-chain-here..."] + } + ] + } +``` + +!!! tip + To get your trust bundle: + + ```bash + # Export your SPIRE server trust bundle + spire-server bundle show -format spiffe > your-trust-bundle.json + ``` + +### Step 2: Submit Federation Request + +1. **Fork the repository**: Go to https://github.com/agntcy/dir and click "Fork" + +2. **Create your federation file**: + ```bash + git clone https://github.com/your-username/dir.git + cd dir/deployment/onboarding/ + cp spire.template.yaml your-org.com.yaml + # Edit your-org.com.yaml with your details + ``` + +3. **Submit a Pull Request**: + - Title: `federation(): add [Your Organization]`. + - Description: Brief description of your organization and use case. + - Files: Include your completed federation configuration. + +### Step 3: Configure Your SPIRE Server + +Add the Directory SPIRE server as a federation peer in your SPIRE server configuration +by obtaining the [Directory trust bundle](https://github.com/agntcy/dir-staging/tree/feat/deploy/onboarding). + +Save the trust bundle to the specified path. + +### Step 4: Verify Federation + +```bash +# Check federation status +spire-server federation list + +# Should show federated trust domain +spire-server federation show --trustDomain dir.agntcy.org +``` + +## Use Cases + +You can find various usage examples in the [Usage Scenarios](./scenarios.md) section. + +## Troubleshooting + +### Connection Issues + +**Problem**: Cannot connect to Directory API +```bash +# Check SPIRE agent status +spire-agent api fetch x509-svid + +# Verify network connectivity +curl -v https://api.directory.agntcy.org + +# Check client configuration +dirctl config list +``` + +### Federation Issues + +**Problem**: SPIRE federation not working +```bash +# Verify trust bundle exchange +spire-server federation show --trustDomain dir.agntcy.org + +# Test bundle endpoint connectivity +curl https://spire.directory.agntcy.org/ +``` + +### Common Error Messages + +| Error | Solution | +| ----------------------------------------------- | ---------------------------------------------------------- | +| `connection refused` | Check if SPIRE agent is running and socket path is correct | +| `x509: certificate signed by unknown authority` | Verify trust bundle configuration | +| `context deadline exceeded` | Check network connectivity and firewall settings | +| `permission denied` | Ensure proper SPIFFE ID registration and policies | + +## Getting Help + +### Community Support + +- **GitHub Issues**: [Open an issue](https://github.com/agntcy/dir/issues) for bugs and feature requests. +- **Discussions**: [GitHub Discussions](https://github.com/agntcy/dir/discussions) for questions and community help. + +For the next steps and getting started, see the [Quick Start Guide](./getting-started.md). For sample applications, see the [Usage Scenarios](./scenarios.md). \ No newline at end of file diff --git a/docs/dir/directory-sandbox-deployment.md b/docs/dir/directory-sandbox-deployment.md new file mode 100644 index 0000000..7ed35f2 --- /dev/null +++ b/docs/dir/directory-sandbox-deployment.md @@ -0,0 +1,317 @@ +# Sandbox Deployment + +This repository contains the deployment manifests for AGNTCY Directory project. +It is designed to be used with Argo CD for GitOps-style continuous deployment. + +The manifests are organized into two main sections: + +- `projects/`: Contains Argo CD project definitions. +- `projectapps/`: Contains Argo CD application definitions. + +The project will deploy the following components: + +- `applications/dir` - AGNTCY Directory server with storage backend. +- `applications/dir-admin` - AGNTCY Directory Admin CLI client. +- `applications/spire*` - SPIRE stack for identity and federation. + +!!! note + This is not a production-ready deployment. It is provided as-is for demonstration and testing purposes. + +## Onboarding + +To onboard a new environment to **Directory Public Staging Network**, check the [onboarding guide](./directory-public-staging.md). + +## Quick Start + +This guide demonstrates how to set up AGNTCY Directory project using Argo CD in Kubernetes [Kind](https://kind.sigs.k8s.io/) cluster. + +1. Create Kind cluster + + ```bash + kind create cluster --name dir-dev + ``` + +2. Install Argo CD in the cluster. + + ```bash + # Install ArgoCD + kubectl create namespace argocd + kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml + + # Wait for ArgoCD to be ready + kubectl wait --namespace argocd --for=condition=available deployment --all --timeout=120s + ``` + +3. Deploy Directory via ArgoCD. + + ```bash + # Add project + kubectl apply -f https://raw.githubusercontent.com/agntcy/dir-staging/main/projects/dir/dev/dir-dev.yaml + + # Add application + kubectl apply -f https://raw.githubusercontent.com/agntcy/dir-staging/main/projectapps/dir/dev/dir-dev-projectapp.yaml + ``` + +4. Check results in ArgoCD UI. + + ```bash + # Retrieve password + kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d; echo + + # Port forward the ArgoCD API to localhost:8080 + kubectl port-forward svc/argocd-server -n argocd 8080:443 + ``` + + Login to the UI at [https://localhost:8080](https://localhost:8080) with username `admin` and the password retrieved above. + + Verify deployment by checking the results of CronJobs in `dir-admin` application. + +5. Clean up. + + ```bash + kind delete cluster --name dir-dev + ``` + +### Token-based Directory Client authentication + +In some cases, you may want to use Directory Client locally +without SPIRE stack. +In this case, you can use token-based authentication +using SPIFFE X509 SVID tokens. + +To generate a SPIFFE SVID token for authenticating local Directory Client +with the Directory Server, follow these steps: + +1. Create a SPIFFE SVID for local Directory Client. + + ```bash + kubectl exec spire-dir-dev-argoapp-server-0 -n dir-dev-spire -c spire-server -- \ + /opt/spire/bin/spire-server x509 mint \ + -dns dev.api.directory.outshift.test \ + -spiffeID spiffe://dev.directory.outshift/local-client \ + -output json > spiffe-dev.json + ``` + +2. Set SPIFFE Token variable for Directory Client. + + ```bash + # Set authentication method to token + export DIRECTORY_CLIENT_AUTH_MODE="token" + export DIRECTORY_CLIENT_SPIFFE_TOKEN="spiffe-dev.json" + + # Set Directory Server address and skip TLS verification + export DIRECTORY_CLIENT_SERVER_ADDRESS="127.0.0.1:8888" + export DIRECTORY_CLIENT_TLS_SKIP_VERIFY="true" + ``` + +3. Port-forward Directory Server API. + + ```bash + kubectl port-forward svc/dir-dir-dev-argoapp-apiserver -n dir-dev-dir 8888:8888 + ``` + +4. Run Directory Client. + + ```bash + dirctl info baeareiesad3lyuacjirp6gxudrzheltwbodtsg7ieqpox36w5j637rchwq + ``` + +## Production Setup + +If you wish to deploy production-grade setup with your own domains and Ingress capabilities on top, follow the steps below. + +!!! note + We are using Minikube to simulate production setup, as it supports Ingress and TLS out of the box. Steps below marked as (local) are optional and intended for local testing purposes only. + +!!! warning + It is not recommended to deploy both dev and prod environments in the same cluster, as they may conflict with each other. + +
+View Production Setup + +
+ +1. Create Minikube cluster. + + ```bash + minikube start -p dir-prod + ``` + +2. (local) Enable Ingress and DNS addons in Minikube. + + The deployment uses `*.test` domain for Ingress resources. + + For local testing purposes, Minikube Ingress controller is required to route traffic to our Ingress resources. + + Otherwise, if you are deploying to a cloud provider with its own Ingress controller, + make sure that it supports SSL Passthrough. + + ```bash + # Enable Ingress and Ingress-DNS addons + minikube addons enable ingress -p dir-prod + minikube addons enable ingress-dns -p dir-prod + + # Patch Ingress controller to enable SSL Passthrough + kubectl patch deployment -n ingress-nginx ingress-nginx-controller --type='json' \ + -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--enable-ssl-passthrough"}]' + ``` + +3. (local) Enable Local DNS inside Minikube. + + The deployment uses `*.test` domain for Ingress resources. + + For local testing purposes, we need to configure DNS resolution + inside Minikube cluster to resolve `*.test` domain to Minikube IP address + using [minikube/ingress-dns](https://minikube.sigs.k8s.io/docs/handbook/addons/ingress-dns) guide. + + Otherwise, if you are deploying to a cloud provider with its own Ingress controller, + you can skip this step. + + ```bash + # Get Minikube IP + minikube ip -p dir-prod + + # Add DNS resolver entry for `*.test` domain + # Follow guide at: https://minikube.sigs.k8s.io/docs/handbook/addons/ingress-dns + + # Update CoreDNS ConfigMap to forward `test` domain to Minikube IP + kubectl edit configmap coredns -n kube-system + ``` + +4. (local) Install CertManager with Self-Signed Issuer. + + The deployment uses CertManager `letsencrypt` issuer to issue TLS certificates for Ingress resources. + + For local testing purposes, we will create a self-signed root CA certificate + and configure CertManager to use it as `letsencrypt` issuer. + + Otherwise, if you are deploying to a cloud provider with its own CertManager, + you can skip this step, but ensure that `letsencrypt` issuer is available. + + ```bash + # Install Cert-Manager + kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.1/cert-manager.yaml + + # Wait for Cert-Manager to be ready + kubectl wait --namespace cert-manager --for=condition=available deployment --all --timeout=120s + + # Create Self-Signed Issuer and Root CA Certificate + kubectl apply -f - < spiffe-prod.json + ``` + +2. Set SPIFFE Token variable for Directory Client. + + ```bash + # Set authentication method to token + export DIRECTORY_CLIENT_AUTH_MODE="token" + export DIRECTORY_CLIENT_SPIFFE_TOKEN="spiffe-prod.json" + + # Set Directory Server address (via Ingress) + export DIRECTORY_CLIENT_SERVER_ADDRESS="prod.api.directory.outshift.test:443" + + # Or, set Directory Server address and skip TLS verification (via port-forwarding) + export DIRECTORY_CLIENT_SERVER_ADDRESS="127.0.0.1:8888" + export DIRECTORY_CLIENT_TLS_SKIP_VERIFY="true" + ``` + +3. Port-forward Directory Server API. + + ```bash + kubectl port-forward svc/dir-dir-prod-argoapp-apiserver -n dir-prod-dir 8888:8888 + ``` + +4. Run Directory Client. + + ```bash + dirctl info baeareiesad3lyuacjirp6gxudrzheltwbodtsg7ieqpox36w5j637rchwq + ``` + +