Instrumented container reachability

Learn how to derive advanced OS package reachability when your container depends on complex external services.

This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Learn how to derive advanced OS package reachability when your container depends on complex external services.

Beta

Instrumented container reachability is an advanced OS package reachability mode that embeds a runtime sensor in your container image to record how your application uses the image in a real environment. Use it when your container relies on complex external services that you cannot exercise during a short local run, so that reachability results then reflect actual usage.

Use instrumented reachability when:

The container has complex external dependencies such as databases, message queues, third-party services that you cannot realistically exercise in a local ephemeral run.
You want profiling to happen in a realistic environment, such as staging, that mirrors production traffic.
You already have integration or end-to-end tests and want reachability to reflect those tests.

You can run instrumented container reachability using either Kubernetes or Docker.

Note

Run the container scan using the new endorctl container scan command. The endorctl scan --container command does not support container reachability.

Prerequisites

To perform instrumented container reachability analysis, ensure that:

Enable container scanning using the --os-reachability flag.
Install and authenticate endorctl.
You have installed Docker daemon dockerd on the host and it runs and is accessible to the current user without elevated privileges. For example, docker images should work without sudo.
The negotiated Docker API version between the client and server is 1.48 or higher.
You must run the scan from either a Linux or a macOS host machine. Endor Labs supports container reachability on both amd64 and arm64 architectures.

To run the instrumented container images, you need either Docker or Kubernetes, based on your setup:

Docker: Docker daemon and Docker CLI is available and you can run the instrumented image locally.
Kubernetes: You have configured kubectl with access to your cluster so you can deploy and run the instrumented image in a pod.

Determine instrumented container reachability using Kubernetes

Follow these steps to scan the original image, collect runtime profiling data, and determine reachability for containers using Kubernetes.

Run endorctl container instrument to create a new image with a lightweight sensor injected into the filesystem. The resulting image will have -instrumented appended to the tag.
```
endorctl container instrument \
  --image=<original_image>:<tag> \
  --app-stop-signal=QUIT \
  --load-instrumented-image=true
```
- --image: Original container image to instrument.
- --app-stop-signal: Signal used to stop the application. The sensor needs this so it can flush profiling data before the container exits.
- --load-instrumented-image: Loads the instrumented image into your local Docker runtime so Kubernetes can reference it.
- To instrument a multi-arch image for specific platforms, run endorctl container instrument with the --platform flag.
```
endorctl container instrument \
  --image=<image_name-tag> \
  --app-stop-signal=QUIT \
  --load-instrumented-image=true \
  --platform=linux/arm64,linux/amd64
```
Define how the instrumented image runs in a manifest. Create a manifest file such as demo-manifest-file.yaml to identify your workload. In the manifest, reference the instrumented image from step 1 and use a pod name and container name you can reuse later. You can also add env, volumes, and other options as needed for your application.
```
apiVersion: v1
kind: Pod
metadata:
  name: <pod-name>
spec:
  restartPolicy: OnFailure
  containers:
    - name: <container-name>
      image: <instrumented-image>
      ports:
        - containerPort: <container-port>
          hostPort: <host-port>
      securityContext:
        privileged: true
```
- Set securityContext.privileged to true so the profiling sensor can run.
- Set restartPolicy to OnFailure so that the pod does not restart automatically after you stop the app to generate the report.
Deploy the instrumented image to Kubernetes. The application runs normally while the sensor observes file access and process activity while the application runs.
```
kubectl apply -f <manifest-file>
kubectl get pods <pod-name>
```
Replace <manifest-file>, <pod-name>, and <container-name> with the values from your manifest.
- If you need to access the application locally, run:
```
kubectl port-forward pod/<pod-name> <host-port>:<container-port>
```
- You can also run your tests or interact with the application normally. The profiling sensor will capture runtime activity.
After you finish testing, send the --app-stop-signal, for example, QUIT, to stop the application gracefully. This signal triggers the profiling sensor to write the creport.json file and generate the profiling data.
```
kubectl exec -it <pod-name> -c <container-name> -- sh -c "kill -QUIT 1"
```
The sensor writes a profiling report to a known artifacts directory inside the container.
- Verify that the creport.json file exists in the container:
```
kubectl exec -it <pod-name> -c <container-name> -- sh -c "ls -ls /opt/_instrumented/artifacts"
```
Create a local directory and copy the report to it.
```
# Create output directory
mkdir -p collect_output
# Copy the profiling data
kubectl cp <pod-name>:/opt/_instrumented/artifacts/creport.json collect_output/creport.json -c <container-name>
```
- To verify the copy, run:
```
ls -la collect_output/
```
- Alternatively, you can use endorctl container collect to stop the running application and retrieve the profiling report from the instrumented container into a local directory. Skip steps 4, 5, and 6 if you are using this command.
```
endorctl container collect \
  --dynamic-profiling-data=true \
  --output-dir=collect_output \
  --image=<instrumented-image>
```
- Set --dynamic-profiling-data to true to collect profiling data from the instrumented container.
- Set --output-dir to the local directory that stores the collected data. The command creates a subdirectory under this path cluster/pod/container. Use that path for --profiling-data-dir in the next step.
Run endorctl container scan with the path to the directory that contains the collected profiling data, and OS reachability enabled. Endor Labs loads the report, maps runtime files to OS packages, and marks the corresponding packages as reachable.
```
endorctl container scan \
  --image=<original_image>:<tag> \
  --profiling-data-dir=collect_output \
  --project-name=<project-name> \
  --os-reachability
```
Remove the pod after completing the analysis:
```
kubectl delete pod <pod-name>
```
Optionally, publish the instrumented image to a registry using the --publish flag. The command pushes the image only if the Docker daemon is already authenticated with the target registry. endorctl will not attempt to re-authenticate with the container registry.
```
endorctl container instrument \
  --image=<image_name-tag> \
  --app-stop-signal=QUIT \
  --load-instrumented-image=true \
  --publish=true
```

Determine instrumented container reachability using Docker

Follow these steps to scan the original image, collect runtime profiling data, and determine reachability for containers using Docker.

Run endorctl container instrument to create a new image with a lightweight sensor injected into the filesystem. At runtime, the sensor writes creport.json to /opt/_instrumented/artifacts/. By default, the command saves the image as instrumented-image.tar. To use a different output path, pass --output-image-tar.
```
endorctl container instrument \
  --image <original_image>:<tag> \
  --load-instrumented-image \
  --app-stop-signal <SIGNAL>
```
- --image: Container image you want to instrument.
- --app-stop-signal: Signal that stops the application so the sensor can flush profiling data before the container exits.
- --load-instrumented-image: Loads the instrumented image into Docker so you can run it with docker run.
Run the instrumented image in Docker with --privileged so the sensor can use ptrace to watch file access. Docker blocks ptrace in unprivileged containers by default. You can run tests or interact with the application while the container runs. The sensor observes file access and process activity.
```
docker run -d \
  --name <container-name> \
  --privileged \
  <original_image>:<tag>-instrumented
```
You can also pass:
- -e KEY=VALUE, for example -e PASSWORD=testpassword123, to provide environment variables your app needs to start.
- -p <host-port>:<container-port>, for example -p 8080:8080, to expose ports when you send traffic from your host during profiling.
After testing, send the --app-stop-signal to stop the application gracefully. This signal triggers the profiling sensor to write the creport.json file. The sleep gives the sensor time to finish writing.
```
docker kill --signal=<SIGNAL> <container-name>
sleep 10
```

Create a local directory and copy the report to it.

mkdir -p ./profile-data
docker cp <container-name>:/opt/_instrumented/artifacts/creport.json ./profile-data/

Run endorctl container scan with the path to the directory that contains the collected profiling data, and OS reachability enabled. Endor Labs loads the report, maps runtime files to OS packages, and marks the corresponding packages as reachable. If you run the command outside a Git repository, pass --project-name <project-name> to avoid an initialization error.
```
endorctl container scan \
  --image <original_image>:<tag> \
  --profiling-data-dir ./profile-data \
  --os-reachability \
  -n <your-namespace>
```
Remove the container after you complete the analysis.
```
docker rm <container-name>
```

Instrumented reachability options

You can run the endorctl container instrument command with the following options.

Flag	Environment Variable	Type	Description
`app-stop-signal`	`ENDOR_CONTAINER_INSTRUMENT_APP_STOP_SIGNAL`	string	Signal sent to the app so the sensor can flush profiling data before the container exits, for example, `QUIT` or `TERM`. Ensure the signal is compatible with your application.
`app-stop-grace-period`	`ENDOR_CONTAINER_INSTRUMENT_APP_STOP_GRACE_PERIOD`	string	Grace period for app shutdown, for example `10s`, `1m`. Use when the app needs time to flush before exit.
`app-stderr-to-file`	`ENDOR_CONTAINER_INSTRUMENT_APP_STDERR_TO_FILE`	boolean	Redirect application error output to a file in the instrumented container.
`app-stdout-to-file`	`ENDOR_CONTAINER_INSTRUMENT_APP_STDOUT_TO_FILE`	boolean	Redirect application standard output to a file in the instrumented container.
`entrypoint`	`ENDOR_CONTAINER_INSTRUMENT_ENTRYPOINT`	string	Override the image entrypoint (JSON array or shell string). Use when the image has a custom entrypoint.
`cmd`	`ENDOR_CONTAINER_INSTRUMENT_CMD`	string	Override the image CMD (JSON array or shell string). Use when the image has a custom CMD.
`debug-mode`	`ENDOR_CONTAINER_INSTRUMENT_DEBUG_MODE`	boolean	Enable sensor debug logs for instrumented container.
`load-instrumented-image`	`ENDOR_CONTAINER_INSTRUMENT_LOAD_INSTRUMENTED_IMAGE`	boolean	Load the instrumented image into the local Docker daemon so Kubernetes or a registry can use it. Default `false`.
`output-image-tar`	`ENDOR_CONTAINER_INSTRUMENT_OUTPUT_IMAGE_TAR`	string	Output tar path for the instrumented image. Default `instrumented-image.tar`.
`sensor-path`	`ENDOR_CONTAINER_INSTRUMENT_SENSOR_PATH`	string	Path to sensor binary.
`platform`	`ENDOR_CONTAINER_INSTRUMENT_PLATFORM`	string	Target platform for the instrumented image, for example `linux/amd64` or `linux/arm64`, or a comma-separated list such as `linux/arm64,linux/amd64` for multi-arch. Use when instrumenting multi-arch images. Requires `--image`.
`publish`	`ENDOR_CONTAINER_INSTRUMENT_PUBLISH`	boolean	Publish the instrumented image to the registry after instrumentation completes. Requires you to authenticate the Docker daemon with the target registry. Default `false`.

You can run the endorctl container collect command with the following options.

Flag	Environment Variable	Type	Description
`dynamic-profiling-data`	`ENDOR_CONTAINER_COLLECT_DYNAMIC_PROFILING_DATA`	boolean	Collect dynamic profiling data from instrumented containers (default `true`).
`kubeconfig-context`	`ENDOR_CONTAINER_COLLECT_KUBECONFIG_CONTEXT`	string	Provide the kubectl kubeconfig context to use to access the target (k8s) deployment environments.
`kubeconfig-path`	`ENDOR_CONTAINER_COLLECT_KUBECONFIG_PATH`	string	Provide the kubectl kubeconfig path to use to access the target (k8s) deployment environments.
`output-dir`	`ENDOR_CONTAINER_COLLECT_OUTPUT_DIR`	string	Set the directory to store collected data from the target deployment environment. The command creates a subdirectory `cluster/pod/container`. Use that path for `--profiling-data-dir` in the scan step.
`runtime-type`	`ENDOR_CONTAINER_COLLECT_RUNTIME_TYPE`	string	Container runtime type. Supports `k8s` only.

Troubleshoot issues

Why is profiling data not generated?

Ensure that you send the QUIT signal correctly.
Check that the container has privileged: true in security context.
Verify that the --app-stop-signal matches the signal your application handles.

Why can’t Kubernetes find my instrumented image?

Run kind load docker-image <image> to load the image into kind.
Push the image to a container registry.

Why does my pod require privileged access?

The profiling sensor uses ptrace to watch file access, which unprivileged containers block by default. Set securityContext.privileged: true in the pod manifest.