Health Monitoring for Private Locations¶

Verbosity Control¶

The Private Location supports configurable log verbosity levels through the ESCAPE_VERBOSITY environment variable. This setting controls the amount of diagnostic information output by the Private Location service.

Available Verbosity Levels¶

Configuration Example¶

The verbosity level can be set using the ESCAPE_VERBOSITY environment variable:

Docker Compose:

services:
  private-location:
    image: escapetech/cli:latest
    restart: always
    command: locations start -v location-name
    environment:
      - ESCAPE_API_KEY=<ESCAPE_API_KEY>
      - ESCAPE_VERBOSITY=1

Helm:

container:
  env:
    - name: ESCAPE_VERBOSITY
      value: "1"

How does Escape determine if a Private Location is alive and operational?¶

Escape monitors Private Location health through a multi-layered approach:

• Regular heartbeats: The Private Location sends periodic health checks to the Escape platform (approximately every second) to signal operational status
• Connection status tracking: The platform continuously monitors the connection state and availability of each Private Location
• Performance monitoring: Escape tracks response times and success rates of scan requests to detect performance degradation

Private Location health and performance can be monitored through the Escape platform's Private Locations dashboard, and additional monitoring can be implemented using the logging and monitoring capabilities.

Can automatic restarts be configured if the Private Location disconnects?¶

Yes, automatic restart on disconnection is supported and can be configured based on the deployment method:

Helm deployment (recommended): By default, Helm deployments include automatic restart functionality. This behavior can be customized by updating the livenessProbe configuration in the Helm values. Other deployment methods: The HEALTH_CHECK_PORT environment variable should be set to a port number greater than 1000 (the container should not run as root). Once configured, custom monitoring can be implemented by querying the /health endpoint and triggering restarts as needed based on the health status.

Advanced Request Logging and Monitoring for Private Locations¶

The CLI can't see the HTTPS/mTLS traffic, That's why we need to use a proxy to intercept the traffic. Below is an example using mitmproxy

Example: Using mitmproxy to extract the X-Escape-Request-Id header¶

You can provide a custom python script to catch the request ID and forward it to the Escape platform. Create a file called ./mitmproxy/extract_escape_request_id.py with the following content:

import requests
import os


class Addon:
    def __init__(self):
        port = os.getenv("HEALTH_CHECK_PORT", "8080")
        self.log_url = f"http://127.0.0.1:{port}/log"

    def request(self, flow):
        request_id = flow.request.headers.get("X-Escape-Request-Id", "")
        if request_id:
            requests.post(self.log_url, data=f'Forwarding X-Escape-Request-Id: {request_id}')

addons = [Addon()]

Then to configure a Private Location to use this addon, you can use the following docker-compose file:

services:
  private-location:
    image: escapetech/cli:latest
    restart: always
    command: locations start -v location-name
    environment:
      - ESCAPE_API_KEY=<ESCAPE_API_KEY>
      - ESCAPE_BACKEND_PROXY_URL=mitm-proxy:8080
      - ESCAPE_ENABLE_LOGS_ENDPOINT=true
  mitm-proxy:
    image: mitmproxy/mitmproxy:latest
    restart: always
    ports:
      - "8080:8080"
    command: "mitmdump -s /mitmproxy/extract_escape_request_id.py"
    volumes:
      - ./mitmproxy:/mitmproxy