Troubleshooting

Solutions for common issues and debugging techniques.

Table of contents

  1. Startup Issues
    1. macOS: “Cannot Be Opened” or “Unverified Developer”
    2. Server Won’t Start
    3. TLS Certificate Errors
  2. Authentication Issues
    1. 401 Unauthorized
    2. 403 Forbidden
  3. Storage Issues
    1. Local Filesystem Errors
    2. S3 Access Denied
    3. GCS Permission Denied
    4. Azure Blob Access Denied
    5. Local Storage Full
  4. Catalog Issues
    1. Table Not Found
    2. Commit Conflict
    3. Namespace Already Exists
  5. Performance Issues
    1. High Latency
    2. Memory Usage High
  6. KMS Issues
    1. AWS KMS Access Denied
    2. Vault Connection Failed
    3. GCP KMS Permission Denied
    4. Azure Key Vault Access Denied
  7. Client Issues
    1. PyIceberg Connection Failed
    2. Trino Connection Failed
  8. Debugging Tools
    1. Enable Debug Logging
    2. Health Checks
    3. Audit Logs
    4. Request Tracing
  9. Getting Help
    1. Collect Diagnostics
    2. Support Channels
  10. Next Steps

Startup Issues

macOS: “Cannot Be Opened” or “Unverified Developer”

Symptom: macOS blocks the binary with a security warning when trying to run it.

"rustberg-darwin-aarch64" cannot be opened because the developer cannot be verified.

Solution: Remove the quarantine attribute from the downloaded binary:

# Remove quarantine attribute
xattr -cr ./rustberg-darwin-aarch64

# Make executable
chmod +x ./rustberg-darwin-aarch64

# Now run it
./rustberg-darwin-aarch64

This is required because the binary was downloaded from the internet and macOS Gatekeeper quarantines it by default.

Server Won’t Start

Symptom: Server exits immediately after starting.

Check 1: Port already in use

# Find process using port
lsof -i :8181

# Use different port
./rustberg --port 8182

Check 2: Storage permissions

# Local storage
ls -la /var/lib/rustberg
chmod 700 /var/lib/rustberg

# S3 permissions
aws sts get-caller-identity
aws s3 ls s3://my-bucket/

Check 3: Config file syntax

# Validate TOML
./rustberg --config config.toml 2>&1 | head -20

TLS Certificate Errors

Symptom: error: failed to load TLS certificate

# Check certificate validity
openssl x509 -in cert.pem -text -noout

# Check key matches certificate
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# Both should match

# Generate new self-signed cert
./rustberg generate-cert --common-name localhost

Authentication Issues

401 Unauthorized

Symptom: All requests return 401.

Check 1: API key format

# Correct format
curl -H "Authorization: Bearer rustberg_xxxxx" \
     http://localhost:8181/v1/config

# Common mistakes:
# ❌ curl -H "Authorization: rustberg_xxxxx"  # Missing "Bearer"
# ❌ curl -H "Bearer rustberg_xxxxx"          # Missing "Authorization:"
# ❌ curl -H "Authorization: Bearer  rustberg_xxxxx"  # Extra space

Check 2: API key validity

# Check if key was rotated or expired
# Verify in audit logs
grep "auth_failure" /var/log/rustberg/audit.log | tail -10

Check 3: JWT configuration

# Test JWKS endpoint
curl https://auth.example.com/.well-known/jwks.json | jq

# Decode JWT to check claims
echo 'eyJhbGc...' | cut -d. -f2 | base64 -d | jq

# Verify issuer and audience match config

403 Forbidden

Symptom: Authentication succeeds but authorization fails.

Check 1: Cedar policy

# Test policy with cedar CLI
cedar evaluate \
  --policies /etc/rustberg/policies/catalog.cedar \
  --principal 'User::"user@example.com"' \
  --action 'Action::"read"' \
  --resource 'Table::"analytics.events"'

Check 2: Tenant isolation

# Verify tenant_id in JWT/API key matches resource
# Check audit log for specific denial reason
grep "authz_deny" /var/log/rustberg/audit.log | tail -10 | jq

Check 3: Role assignment

# Verify user has correct roles
# Check API key metadata or JWT claims

Storage Issues

Local Filesystem Errors

Symptom: read-only filesystem or storage medium error when using file:// warehouse.

opendal::layers::retry: will retry after 2s because: Unexpected (temporary) at write, 
context: { service: fs, path: warehouse/... } => read-only filesystem or storage medium, 
source: Read-only file system (os error 30)

Cause: The warehouse directory doesn’t exist or the path is malformed.

Solution: Rustberg automatically creates local directories and supports relative paths:

# Relative path (creates ./warehouse in current directory)
./rustberg --warehouse file://warehouse

# Absolute path
./rustberg --warehouse file:///var/lib/rustberg/warehouse

# Bare relative path also works
./rustberg --warehouse warehouse

Rustberg automatically converts relative paths to absolute paths and creates the directory if it doesn’t exist.

Check directory permissions:

# Verify the resolved path
ls -la $(pwd)/warehouse

# If permission denied, fix ownership
sudo chown -R $(whoami) /path/to/warehouse

S3 Access Denied

Symptom: AccessDenied errors when accessing S3.

# Verify credentials
aws sts get-caller-identity

# Test bucket access
aws s3 ls s3://my-bucket/rustberg-catalog/

# Check bucket policy
aws s3api get-bucket-policy --bucket my-bucket

# Verify IAM permissions
aws iam get-user-policy --user-name myuser --policy-name rustberg

GCS Permission Denied

Symptom: 403 Forbidden when accessing GCS.

# Verify service account
gcloud auth list

# Test bucket access
gsutil ls gs://my-bucket/rustberg-catalog/

# Check IAM binding
gsutil iam get gs://my-bucket

Azure Blob Access Denied

Symptom: AuthorizationFailure when accessing Azure.

# Verify credentials
az account show

# Test container access
az storage blob list \
  --account-name mystorageaccount \
  --container-name mycontainer

# Check RBAC assignment
az role assignment list --assignee <service-principal-id>

Local Storage Full

Symptom: No space left on device

# Check disk usage
df -h /var/lib/rustberg

# Clean old data (if safe)
du -sh /var/lib/rustberg/*

# Consider moving to larger disk or cloud storage

Catalog Issues

Table Not Found

Symptom: NoSuchTableException for existing table.

Check 1: Correct namespace

# List namespaces
curl -H "Authorization: Bearer $API_KEY" \
     http://localhost:8181/v1/namespaces | jq

# List tables in namespace
curl -H "Authorization: Bearer $API_KEY" \
     http://localhost:8181/v1/namespaces/my_namespace/tables | jq

Check 2: Tenant isolation

# Ensure API key has correct tenant_id
# Tables are isolated by tenant

Commit Conflict

Symptom: CommitFailedException on table update.

This is normal behavior with optimistic concurrency. Solutions:

  1. Retry with backoff: 
    from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential())
def update_table():
    # Your update code
  2. Check for concurrent writers:
    • Multiple processes updating same table
    • Consider serializing updates

Namespace Already Exists

Symptom: AlreadyExistsException when creating namespace.

# Check if namespace exists
curl -H "Authorization: Bearer $API_KEY" \
     http://localhost:8181/v1/namespaces/my_namespace | jq

# If exists, use it or choose different name

Performance Issues

High Latency

Symptom: Requests take longer than expected.

Check 1: Network latency

# Test from client to server
curl -w "@curl-format.txt" -o /dev/null -s \
     http://localhost:8181/health

# curl-format.txt:
# time_namelookup:  %{time_namelookup}s\n
# time_connect:     %{time_connect}s\n
# time_appconnect:  %{time_appconnect}s\n
# time_total:       %{time_total}s\n

Check 2: Storage backend latency

# S3 latency
aws s3api head-object \
  --bucket my-bucket \
  --key rustberg-catalog/test

Check 3: Rate limiting

# Check for rate limit headers
curl -i -H "Authorization: Bearer $API_KEY" \
     http://localhost:8181/v1/config | grep -i ratelimit

Memory Usage High

Symptom: Memory usage exceeds expected ~9MB.

# Check actual usage
ps aux | grep rustberg

# Check for memory leaks
# Monitor over time
watch -n 5 'ps aux | grep rustberg'

Possible causes:

  • Large number of cached DEKs (encryption)
  • Many concurrent connections
  • Large request bodies

KMS Issues

AWS KMS Access Denied 

# Verify KMS permissions
aws kms describe-key --key-id <key-id>

# Test encrypt/decrypt
aws kms encrypt \
  --key-id <key-id> \
  --plaintext "test" \
  --output text --query CiphertextBlob

Vault Connection Failed 

# Test Vault connectivity
curl $VAULT_ADDR/v1/sys/health

# Verify token
vault token lookup

# Test transit engine
vault read transit/keys/rustberg-key

GCP KMS Permission Denied 

# Test KMS permissions
gcloud kms keys describe rustberg-key \
  --keyring=rustberg-keyring \
  --location=global

Azure Key Vault Access Denied 

# Test Key Vault access
az keyvault key show \
  --vault-name rustberg-vault \
  --name rustberg-key

Client Issues

PyIceberg Connection Failed 

# Debug connection
import logging
logging.basicConfig(level=logging.DEBUG)

from pyiceberg.catalog import load_catalog
catalog = load_catalog("rustberg", uri="http://localhost:8181")

Trino Connection Failed 

-- Check catalog status
SHOW CATALOGS;

-- If missing, check connector config
-- catalog/rustberg.properties

Debugging Tools

Enable Debug Logging 

# Via environment variable
RUSTBERG_LOG_LEVEL=debug ./rustberg

# Via config
[logging]
level = "debug"

Health Checks 

# Liveness
curl http://localhost:8181/health

# Readiness
curl http://localhost:8181/ready

# Metrics
curl http://localhost:8181/metrics

Audit Logs 

# View recent auth events
tail -f /var/log/rustberg/audit.log | jq

# Filter by event type
grep "authz_deny" audit.log | jq

Request Tracing 

# Include request ID in all requests
curl -H "X-Request-Id: debug-123" \
     -H "Authorization: Bearer $API_KEY" \
     http://localhost:8181/v1/namespaces

# Find in logs
grep "debug-123" /var/log/rustberg/app.log

Getting Help

Collect Diagnostics

Before opening an issue, collect:

  1. Rustberg version: 
    ./rustberg --version
  2. Configuration (sanitized): 
    cat config.toml | grep -v -E "(key|secret|password|token)"
  3. Error logs: 
    tail -100 /var/log/rustberg/app.log
  4. System info: 
    uname -a

Support Channels

Next Steps