ASLD v0.1.0 Timeout Error - Cannot Connect to Aerospike Server

[khushboojaiswal-ux]

Aerospike Bug Report: ASLD Timeout Issue

Summary

ASLD (Aerospike Link Daemon) v0.1.0 consistently times out when attempting to connect to Aerospike server, regardless of server version.

Environment

• ASLD Version: 0.1.0 (from Aerospike PHP Client v1.3.0)
• PHP Client Version: v1.3.0
• PHP Version: 8.2
• Aerospike Server Versions Tested:
  • aerospike/aerospike-server:6.4.0.4 ❌ Timeout
  • aerospike/aerospike-server:latest ❌ Timeout
• Operating System: Linux (Docker container based on php:8.2-apache)
• Architecture: x86_64

Problem Description

ASLD starts successfully but immediately times out when attempting to connect to the Aerospike server. The process exits, and the Unix socket (/tmp/asld_grpc.sock) is never created.

Error Message

2025/11/21 07:25:00 Aerospike Local Proxy `0.1.0`.
2025/11/21 07:25:00 ResultCode: TIMEOUT, Iteration: 0, InDoubt: false, Node: <nil>: command execution timed out on client: See `Policy.Timeout`

Symptoms

1. ASLD process starts but exits immediately after timeout
2. Unix socket /tmp/asld_grpc.sock is never created
3. PHP extension cannot connect (gets "transport error")
4. Issue occurs with multiple Aerospike server versions
5. Network connectivity to Aerospike server is confirmed working

Configuration

ASLD Configuration (asld.toml)

[server]
listen = "unix:///tmp/asld_grpc.sock"

[client]
hosts = ["aerospike-server:3000"]
timeout = "120s"
login-timeout = "30s"
fail-if-not-connected = false

Docker Compose

services:
  aerospike-server:
    image: aerospike/aerospike-server:latest
    container_name: aerospike-server
    ports:
      - "3000:3000"
      - "3001:3001"
      - "3002:3002"
    volumes:
      - aerospike-data:/opt/aerospike/data
    environment:
      SERVICE_NAME: aerospike
      NAMESPACE: test
      STORAGE: /opt/aerospike/data/test.dat
      STORAGE_SIZE: 4G

  laravel-app:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: laravel-app
    ports:
      - "8081:80"
    depends_on:
      - aerospike-server

Steps to Reproduce

1. Build Aerospike PHP Client v1.3.0 from source
2. Build ASLD from aerospike-connection-manager/ directory
3. Configure ASLD with asld.toml pointing to Aerospike server
4. Start Aerospike server (any version tested)
5. Start ASLD: /usr/local/bin/asld -config-file /etc/aerospike-connection-manager/asld.toml
6. Observe timeout error in logs
7. Verify socket is not created: ls -la /tmp/asld_grpc.sock

Expected Behavior

1. ASLD should connect to Aerospike server successfully
2. Unix socket should be created at /tmp/asld_grpc.sock
3. PHP extension should be able to connect via the socket
4. No timeout errors

Actual Behavior

1. ASLD times out immediately when trying to connect
2. Process exits
3. Socket is never created
4. PHP extension cannot connect

Verification Steps Performed

Network Connectivity

# From laravel-app container
timeout 2 bash -c 'cat < /dev/null > /dev/tcp/aerospike-server/3000'
# Result: ✅ Connection successful

Server Status

docker-compose ps
# Result: ✅ Both containers running
# - aerospike-server: Up and running
# - laravel-app: Up and running

ASLD Process

ps aux | grep asld
# Result: ❌ Process not running (exited after timeout)

Socket File

ls -la /tmp/asld_grpc.sock
# Result: ❌ File does not exist

Additional Information

Build Process

ASLD was built using:

cd aerospike-connection-manager
go mod download
go build -o asld .

PHP Extension

PHP extension was built successfully and is loaded:

php -m | grep aerospike
# Result: aerospike_php

Workaround

Currently using Laravel cache as fallback when Aerospike connection fails.

Impact

• Severity: High - ASLD is required for modern PHP client
• Workaround Available: Yes (cache fallback)
• Production Ready: No - Cannot use Aerospike with PHP extension

Where to Report

• GitHub: https://github.com/aerospike/php-client/issues
• Aerospike Forum: https://discuss.aerospike.com/
• Aerospike Support: (if you have support contract)

Additional Files

• ASLD configuration: docker/asld.toml
• Startup script: docker/start.sh
• Dockerfile: Dockerfile
• Full ASLD log available upon request

Contact Information

(Add your contact details here if reporting directly)

---

Report Date: 2025-11-21
Status: Unresolved
Priority: High

[khaf]

Thank you for this thorough report and sorry your report flew under my radar. Does the user running the asld service have access to create the socket?

[khushboojaiswal-ux]

Thank you for the quick response! Yes, I've verified socket permissions. Here are the details:

User Running ASLD

Container Base: php:8.2-apache (Docker)

ASLD Execution:

• Initially tried running ASLD as the default container user (likely www-data)
• Also tested running as root user explicitly
• Both scenarios resulted in the same timeout error

Socket Permissions Check

Directory Permissions:

# Checked /tmp directory permissions
ls -ld /tmp
# Result: drwxrwxrwt (1777) - standard tmpfs permissions, writable by all

# Checked if socket file exists
ls -la /tmp/asld_grpc.sock
# Result: No such file or directory (socket never created)

User Information:

# When running ASLD manually
whoami
# Result: root (when tested as root)

# Or
id
# Result: uid=0(root) gid=0(root) groups=0(root)

Permission-Related Observations

1. No Permission Errors in Logs:

   • ASLD logs show only timeout errors
   • No "Permission denied" or "Cannot create socket" errors
   • The timeout occurs before socket creation is attempted

2. Socket Path Verification:

   • /tmp is writable (standard tmpfs with 1777 permissions)
   • Tried with explicit permissions: chmod 777 /tmp (no change)
   • Tried creating socket manually: touch /tmp/asld_grpc.sock (works fine)

3. ASLD Process Behavior:

   • ASLD process starts successfully
   • Immediately attempts connection to Aerospike server
   • Times out during connection attempt
   • Process exits before socket creation

Additional Diagnostic Information

Network Connectivity (Confirmed Working):

# From laravel-app container
timeout 2 bash -c 'cat < /dev/null > /dev/tcp/aerospike-server/3000'
# Result: ✅ Connection successful (no timeout)

# Direct connection test
nc -zv aerospike-server 3000
# Result: ✅ Connection successful

ASLD Configuration:

[server]
listen = "unix:///tmp/asld_grpc.sock"

[client]
hosts = ["aerospike-server:3000"]
timeout = "120s"
login-timeout = "30s"
fail-if-not-connected = false

Full ASLD Log Output:

2025/11/21 07:25:00 Aerospike Local Proxy `0.1.0`.
2025/11/21 07:25:00 ResultCode: TIMEOUT, Iteration: 0, InDoubt: false, Node: <nil>: command execution timed out on client: See `Policy.Timeout`

Conclusion

The timeout occurs before socket creation is attempted. The issue appears to be:

1. ASLD successfully starts
2. ASLD attempts to connect to Aerospike server at aerospike-server:3000
3. Connection times out (despite network connectivity being confirmed)
4. ASLD exits without creating the socket

This suggests the issue is with the connection logic in ASLD, not socket permissions. The socket would be created after a successful connection, but ASLD never reaches that point.

Questions

1. Is there a specific order of operations ASLD should follow? (Connect first, then create socket?)
2. Are there any known issues with ASLD v0.1.0 connecting to Aerospike server versions 6.4.0.4 or 8.1.0.1?
3. Should we try a different connection method or configuration?

Workaround Status

Currently using Aerospike REST Gateway as a workaround, which works perfectly. However, we'd prefer to use the native PHP extension if ASLD can be made stable.

---

Quick Diagnostic Commands (If Needed)

If you need me to run additional diagnostics, here are commands I can execute:

# 1. Check current user and permissions
whoami
id
ls -ld /tmp

# 2. Try creating socket manually (to verify permissions)
touch /tmp/asld_grpc.sock
ls -la /tmp/asld_grpc.sock
rm /tmp/asld_grpc.sock

# 3. Run ASLD with verbose logging (if available)
/usr/local/bin/asld -config-file /etc/aerospike-connection-manager/asld.toml -v

# 4. Check if ASLD process is running
ps aux | grep asld

# 5. Check network connectivity from container
nc -zv aerospike-server 3000
telnet aerospike-server 3000

---

Note: The socket permissions are not the issue - the timeout occurs during the connection attempt, before socket creation is attempted.