herocode/herolib_rust
12
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

...

Browse Source
This commit is contained in:
kristof
2025-04-02 08:49:40 +02:00
parent a3c6322b49
commit c7908cb6e4
67 changed files with 11897 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

467
src/virt/nerdctl/nerdctl-essentials.md Normal file
View File

@@ -0,0 +1,467 @@
# nerdctl Essentials
This guide provides a comprehensive overview of essential nerdctl functionality to help you get started quickly. nerdctl is a Docker-compatible CLI for containerd, with additional features specifically designed for containerd environments.
## Introduction
nerdctl is a Docker-compatible CLI for containerd. It provides the same user experience as the Docker CLI (`docker`) but leverages the more efficient containerd container runtime. Key differences and advantages include:
- Direct integration with containerd (no extra daemon required)
- Support for containerd-specific features
- First-class support for rootless mode
- Compatibility with Docker commands
- Additional nerdctl-specific commands
## Basic Configuration
nerdctl can be configured using the `nerdctl.toml` configuration file:
- Rootful mode: `/etc/nerdctl/nerdctl.toml`
- Rootless mode: `~/.config/nerdctl/nerdctl.toml`
Example configuration:
```toml
debug = false
debug_full = false
address = "unix:///run/containerd/containerd.sock"
namespace = "default"
snapshotter = "overlayfs"
cgroup_manager = "systemd"
hosts_dir = ["/etc/containerd/certs.d", "/etc/docker/certs.d"]
```
Common configuration properties:
| Property | CLI Flag | Description |
|---------------------|-----------------------------------|----------------------------|
| `address` | `--address`, `--host`, `-a`, `-H` | containerd address |
| `namespace` | `--namespace`, `-n` | containerd namespace |
| `snapshotter` | `--snapshotter` | containerd snapshotter |
| `cni_path` | `--cni-path` | CNI binary directory |
| `data_root` | `--data-root` | Persistent state directory |
| `insecure_registry` | `--insecure-registry` | Allow insecure registry |
## Container Management
### Running Containers
**Run a container**:
```
nerdctl run [OPTIONS] IMAGE [COMMAND] [ARG...]
```
Common options:
- `-i, --interactive`: Keep STDIN open
- `-t, --tty`: Allocate a pseudo-TTY
- `-d, --detach`: Run container in background
- `--name`: Assign a name to the container
- `-p, --publish`: Publish container's port to the host
- `-v, --volume`: Bind mount a volume
- `-e, --env`: Set environment variables
- `--rm`: Automatically remove the container when it exits
- `--restart=(no|always|on-failure|unless-stopped)`: Restart policy
- `--net, --network`: Connect container to a network
Examples:
```bash
# Run an interactive container and automatically remove it when it exits
nerdctl run -it --rm alpine sh
# Run a detached container with port mapping
nerdctl run -d --name nginx -p 8080:80 nginx
# Run a container with a volume mount
nerdctl run -it --rm -v $(pwd):/data alpine ls /data
```
### Managing Containers
**List containers**:
```
nerdctl ps [OPTIONS]
```
Options:
- `-a, --all`: Show all containers (default shows just running)
- `-q, --quiet`: Only display container IDs
- `-s, --size`: Display total file sizes
**Stop a container**:
```
nerdctl stop [OPTIONS] CONTAINER [CONTAINER...]
```
**Start a container**:
```
nerdctl start [OPTIONS] CONTAINER [CONTAINER...]
```
**Remove a container**:
```
nerdctl rm [OPTIONS] CONTAINER [CONTAINER...]
```
Options:
- `-f, --force`: Force removal of running container
- `-v, --volumes`: Remove anonymous volumes
**View container logs**:
```
nerdctl logs [OPTIONS] CONTAINER
```
Options:
- `-f, --follow`: Follow log output
- `--since`: Show logs since timestamp
- `-t, --timestamps`: Show timestamps
- `-n, --tail`: Number of lines to show from the end of logs
**Execute a command in a running container**:
```
nerdctl exec [OPTIONS] CONTAINER COMMAND [ARG...]
```
Options:
- `-i, --interactive`: Keep STDIN open
- `-t, --tty`: Allocate a pseudo-TTY
- `-d, --detach`: Detached mode
- `-w, --workdir`: Working directory
- `-e, --env`: Set environment variables
## Image Management
### Working with Images
**List images**:
```
nerdctl images [OPTIONS]
```
Options:
- `-a, --all`: Show all images
- `-q, --quiet`: Only show numeric IDs
- `--digests`: Show digests
**Pull an image**:
```
nerdctl pull [OPTIONS] NAME[:TAG|@DIGEST]
```
Options:
- `--platform=(amd64|arm64|...)`: Pull content for specific platform
- `-q, --quiet`: Suppress verbose output
**Push an image**:
```
nerdctl push [OPTIONS] NAME[:TAG]
```
**Build an image**:
```
nerdctl build [OPTIONS] PATH
```
Options:
- `-t, --tag`: Name and optionally tag the image
- `-f, --file`: Name of the Dockerfile
- `--build-arg`: Set build-time variables
- `--no-cache`: Do not use cache when building
**Remove an image**:
```
nerdctl rmi [OPTIONS] IMAGE [IMAGE...]
```
Options:
- `-f, --force`: Force removal
**Tag an image**:
```
nerdctl tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]
```
**Save an image to a tar archive**:
```
nerdctl save [OPTIONS] IMAGE [IMAGE...]
```
Options:
- `-o, --output`: Write to a file
**Load an image from a tar archive**:
```
nerdctl load [OPTIONS]
```
Options:
- `-i, --input`: Read from a tar archive file
## Network Management
### Working with Networks
**List networks**:
```
nerdctl network ls [OPTIONS]
```
**Create a network**:
```
nerdctl network create [OPTIONS] NETWORK
```
Common options:
- `-d, --driver=(bridge|macvlan|ipvlan)`: Driver to manage the network
- `--subnet`: Subnet in CIDR format (e.g., "10.5.0.0/16")
- `--gateway`: Gateway for the subnet
- `--ipam-driver=(default|host-local|dhcp)`: IP address management driver
**Remove a network**:
```
nerdctl network rm NETWORK [NETWORK...]
```
**Inspect a network**:
```
nerdctl network inspect [OPTIONS] NETWORK [NETWORK...]
```
**Prune networks**:
```
nerdctl network prune [OPTIONS]
```
### Network Types
nerdctl supports the following network types:
- `bridge` (default on Linux): Creates a bridge interface on the host
- `host`: Uses the host's network stack
- `none`: No networking
- `macvlan`: Connects container interfaces directly to host interfaces
- `ipvlan`: Similar to macvlan but shares host's IP address
Example creating a macvlan network:
```bash
nerdctl network create macnet --driver macvlan \
--subnet=192.168.5.0/24 \
--gateway=192.168.5.1 \
-o parent=eth0
```
## Volume Management
### Working with Volumes
**List volumes**:
```
nerdctl volume ls [OPTIONS]
```
**Create a volume**:
```
nerdctl volume create [OPTIONS] [VOLUME]
```
**Remove a volume**:
```
nerdctl volume rm [OPTIONS] VOLUME [VOLUME...]
```
**Inspect a volume**:
```
nerdctl volume inspect [OPTIONS] VOLUME [VOLUME...]
```
**Prune volumes**:
```
nerdctl volume prune [OPTIONS]
```
### Volume Flags for Containers
Volume-related flags when running containers:
- `-v, --volume`: Bind mount a volume (format: `SRC:DST[:OPTIONS]`)
- `--mount`: Attach a filesystem mount to the container
- `--tmpfs`: Mount a tmpfs directory
Volume options:
- `rw`: Read/write (default)
- `ro`: Read-only
- `rro`: Recursive read-only (kernel >= 5.12)
- `shared`, `slave`, `private`: Non-recursive propagation
- `rshared`, `rslave`, `rprivate`: Recursive propagation
Examples:
```bash
# Mount a host directory
nerdctl run -it --rm -v /host/path:/container/path:ro alpine ls /container/path
# Use tmpfs
nerdctl run -it --rm --tmpfs /tmp:size=64m,exec alpine ls /tmp
```
## Compose
nerdctl includes Docker Compose compatibility, allowing you to define and run multi-container applications.
**Run Compose applications**:
```
nerdctl compose up [OPTIONS]
```
Options:
- `-d, --detach`: Run containers in the background
- `--build`: Build images before starting containers
- `--no-build`: Don't build images, even if they're missing
- `--force-recreate`: Force recreation of containers
**Stop Compose applications**:
```
nerdctl compose down [OPTIONS]
```
Options:
- `-v, --volumes`: Remove named volumes and anonymous volumes
**View Compose logs**:
```
nerdctl compose logs [OPTIONS] [SERVICE...]
```
Other Compose commands:
- `nerdctl compose build`: Build service images
- `nerdctl compose ps`: List containers
- `nerdctl compose pull`: Pull service images
- `nerdctl compose exec`: Execute a command in a running container
- `nerdctl compose restart`: Restart services
Example `compose.yml`:
```yaml
version: "3.8"
services:
web:
image: nginx
ports:
- "8080:80"
volumes:
- ./html:/usr/share/nginx/html
db:
image: postgres
environment:
POSTGRES_PASSWORD: example
volumes:
- db-data:/var/lib/postgresql/data
volumes:
db-data:
```
## Rootless Mode
nerdctl supports rootless containers, allowing unprivileged users to create and manage containers. This provides better security isolation compared to running everything as root.
### Setup Rootless Mode
1. Install required dependencies (see https://rootlesscontaine.rs/getting-started/common/)
2. Set up rootless containerd:
```
containerd-rootless-setuptool.sh install
```
3. Enable lingering for your user (to keep services running after logout):
```
sudo loginctl enable-linger $(whoami)
```
4. For building images, install BuildKit in rootless mode:
```
containerd-rootless-setuptool.sh install-buildkit
```
When running in rootless mode, nerdctl automatically uses the appropriate socket and configuration.
### Limitations and Considerations
- Resource limits require cgroup v2 and systemd
- By default, ports below 1024 cannot be published (use slirp4netns port driver or configure capabilities)
- Some file system operations might be restricted
- Network performance can be slower (consider using bypass4netns to improve performance)
## Registry Authentication
nerdctl uses the same authentication configuration as Docker, located in `${DOCKER_CONFIG}/config.json` (default: `$HOME/.docker/config.json`).
**Log in to a registry**:
```
nerdctl login [OPTIONS] [SERVER]
```
Options:
- `-u, --username`: Username
- `-p, --password`: Password
- `--password-stdin`: Take the password from stdin
**Log out from a registry**:
```
nerdctl logout [SERVER]
```
### Registry Certificates
For private registries with custom certificates, place certificates in:
- Rootful: `/etc/containerd/certs.d/<HOST:PORT>/` or `/etc/docker/certs.d/<HOST:PORT>/`
- Rootless: `~/.config/containerd/certs.d/<HOST:PORT>/` or `~/.config/docker/certs.d/<HOST:PORT>/`
## Advanced Features
### GPU Support
nerdctl supports NVIDIA GPU passthrough to containers:
```
nerdctl run -it --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu20.04 nvidia-smi
```
Options for `--gpus`:
- `all`: Use all available GPUs
- Custom configuration: `--gpus '"capabilities=utility,compute",device=GPU-UUID'`
### BuildKit Integration
BuildKit provides advanced image building capabilities:
1. Set up BuildKit (different for rootful and rootless):
```
# Rootless with containerd worker
CONTAINERD_NAMESPACE=default containerd-rootless-setuptool.sh install-buildkit-containerd
```
2. Use advanced build features:
```
nerdctl build --output=type=local,dest=./output --platform=linux/amd64,linux/arm64 .
```
### Namespace Management
**Create a namespace**:
```
nerdctl namespace create NAMESPACE
```
**List namespaces**:
```
nerdctl namespace ls
```
**Remove a namespace**:
```
nerdctl namespace remove NAMESPACE
```
### Security Features
nerdctl supports various security features:
- `--security-opt seccomp=profile.json`: Apply a seccomp profile
- `--security-opt apparmor=profile`: Apply an AppArmor profile
- `--cap-add`/`--cap-drop`: Add or drop Linux capabilities
- `--privileged`: Give extended privileges to the container

103
src/virt/nerdctl/nerdctldocs/build.md Normal file
View File

@@ -0,0 +1,103 @@
# Setting up `nerdctl build` with BuildKit
`nerdctl build` (and `nerdctl compose build`) relies on [BuildKit](https://github.com/moby/buildkit).
To use it, you need to set up BuildKit.
BuildKit has 2 types of backends.
- **containerd worker**: BuildKit relies on containerd to manage containers and images, etc. containerd needs to be up-and-running on the host.
- **OCI worker**: BuildKit manages containers and images, etc. containerd isn't needed. This worker relies on runc for container execution.
You need to set up BuildKit with either of the above workers.
Note that OCI worker cannot access base images (`FROM` images in Dockerfiles) managed by containerd.
Thus you cannot let `nerdctl build` use containerd-managed images as the base image.
They include images previously built using `nerdctl build`.
For example, the following build `bar` fails with OCI worker because it tries to use the previously built and containerd-managed image `foo`.
```console
$ mkdir -p /tmp/ctx && cat <<EOF > /tmp/ctx/Dockerfile
FROM ghcr.io/stargz-containers/ubuntu:20.04-org
RUN echo hello
EOF
$ nerdctl build -t foo /tmp/ctx
$ cat <<EOF > /tmp/ctx/Dockerfile
FROM foo
RUN echo bar
EOF
$ nerdctl build -t bar /tmp/ctx
```
This limitation can be avoided using containerd worker as mentioned later.
## Setting up BuildKit with containerd worker
### Rootless
| :zap: Requirement | nerdctl >= 0.18, BuildKit >= 0.10 |
|-------------------|-----------------------------------|
```
$ CONTAINERD_NAMESPACE=default containerd-rootless-setuptool.sh install-buildkit-containerd
```
`containerd-rootless-setuptool.sh` is aware of `CONTAINERD_NAMESPACE` and `CONTAINERD_SNAPSHOTTER` envvars.
It installs buildkitd to the specified containerd namespace.
This allows BuildKit using containerd-managed images in that namespace as the base image.
Note that BuildKit can't use images in other namespaces as of now.
If `CONTAINERD_NAMESPACE` envvar is not specified, this script configures buildkitd to use "buildkit" namespace (not "default" namespace).
You can install an additional buildkitd process in a different namespace by executing this script with specifying the namespace with `CONTAINERD_NAMESPACE`.
BuildKit will expose the socket at `$XDG_RUNTIME_DIR/buildkit-$CONTAINERD_NAMESPACE/buildkitd.sock` if `CONTAINERD_NAMESPACE` is specified.
If `CONTAINERD_NAMESPACE` is not specified, that location will be `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`.
### Rootful
```
$ sudo systemctl enable --now buildkit
```
Then add the following configuration to `/etc/buildkit/buildkitd.toml` to enable containerd worker.
```toml
[worker.oci]
enabled = false
[worker.containerd]
enabled = true
# namespace should be "k8s.io" for Kubernetes (including Rancher Desktop)
namespace = "default"
```
## Setting up BuildKit with OCI worker
### Rootless
```
$ containerd-rootless-setuptool.sh install-buildkit
```
As mentioned in the above, BuildKit with this configuration cannot use images managed by containerd.
They include images previously built with `nerdctl build`.
BuildKit will expose the socket at `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`.
### rootful
```
$ sudo systemctl enable --now buildkit
```
## Which BuildKit socket will nerdctl use?
You can specify BuildKit address for `nerdctl build` using `--buildkit-host` flag or `BUILDKIT_HOST` envvar.
When BuildKit address isn't specified, nerdctl tries some default BuildKit addresses the following order and uses the first available one.
- `<runtime directory>/buildkit-<current namespace>/buildkitd.sock`
- `<runtime directory>/buildkit-default/buildkitd.sock`
- `<runtime directory>/buildkit/buildkitd.sock`
For example, if you run rootless nerdctl with `test` containerd namespace, it tries to use `$XDG_RUNTIME_DIR/buildkit-test/buildkitd.sock` by default then try to fall back to `$XDG_RUNTIME_DIR/buildkit-default/buildkitd.sock` and `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`

166
src/virt/nerdctl/nerdctldocs/cni.md Normal file
View File

@@ -0,0 +1,166 @@
# Using CNI with nerdctl
nerdctl uses CNI plugins for its container network, you can set network by
either `--network` or `--net` option.
## Basic networks
nerdctl support some basic types of CNI plugins without any configuration
needed(you should have CNI plugin be installed), for Linux systems the basic
CNI plugin types are `bridge`, `portmap`, `firewall`, `tuning`, for Windows
system, the supported CNI plugin types are `nat` only.
The default network `bridge` for Linux and `nat` for Windows if you
don't set any network options.
Configuration of the default network `bridge` of Linux:
```json
{
"cniVersion": "1.0.0",
"name": "bridge",
"plugins": [
{
"type": "bridge",
"bridge": "nerdctl0",
"isGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"routes": [{ "dst": "0.0.0.0/0" }],
"ranges": [
[
{
"subnet": "10.4.0.0/24",
"gateway": "10.4.0.1"
}
]
]
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall",
"ingressPolicy": "same-bridge"
},
{
"type": "tuning"
}
]
}
```
## Bridge isolation
nerdctl >= 0.18 sets the `ingressPolicy` to `same-bridge` when `firewall` plugin >= 1.1.0 is installed.
This `ingressPolicy` replaces the CNI `isolation` plugin used in nerdctl <= 0.17.
When `firewall` plugin >= 1.1.0 is not found, nerdctl does not enable the bridge isolation.
This means a container in `--net=foo` can connect to a container in `--net=bar`.
## macvlan/IPvlan networks
nerdctl also support macvlan and IPvlan network driver.
To create a `macvlan` network which bridges with a given physical network interface, use `--driver macvlan` with
`nerdctl network create` command.
```
# nerdctl network create mac0 --driver macvlan \
--subnet=192.168.5.0/24
--gateway=192.168.5.2
-o parent=eth0
```
You can specify the `parent`, which is the interface the traffic will physically go through on the host,
defaults to default route interface.
And the `subnet` should be under the same network as the network interface,
an easier way is to use DHCP to assign the IP:
```
# nerdctl network create mac0 --driver macvlan --ipam-driver=dhcp
```
Using `--driver ipvlan` can create `ipvlan` network, the default mode for IPvlan is `l2`.
## DHCP host-name and other DHCP options
Nerdctl automatically sets the DHCP host-name option to the hostname value of the container.
Furthermore, on network creation, nerdctl supports the ability to set other DHCP options through `--ipam-options`.
Currently, the following options are supported by the DHCP plugin:
```
dhcp-client-identifier
subnet-mask
routers
user-class
vendor-class-identifier
```
For example:
```
# nerdctl network create --driver macvlan \
--ipam-driver dhcp \
--ipam-opt 'vendor-class-identifier={"type": "provide", "value": "Hey! Its me!"}' \
my-dhcp-net
```
## Custom networks
You can also customize your CNI network by providing configuration files.
When rootful, the expected root location is `/etc/cni/net.d`.
For rootless, the expected root location is `~/.config/cni/net.d/`
Configuration files (like `10-mynet.conf`) can be placed either in the root location,
or under a subfolder.
If in the root location, this network will be available to all nerdctl namespaces.
If placed in a subfolder, it will be available only to the identically named namespace.
For example, you have one configuration file(`/etc/cni/net.d/10-mynet.conf`)
for `bridge` network:
```json
{
"cniVersion": "1.0.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "172.19.0.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
```
This will configure a new CNI network with the name `mynet`, and you can use
this network to create a container in any namespace:
```console
# nerdctl run -it --net mynet --rm alpine ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if6120: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 5e:5b:3f:0c:36:56 brd ff:ff:ff:ff:ff:ff
inet 172.19.0.51/24 brd 172.19.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5c5b:3fff:fe0c:3656/64 scope link tentative
valid_lft forever preferred_lft forever
```

1806
src/virt/nerdctl/nerdctldocs/command-reference.md Normal file
View File

File diff suppressed because it is too large Load Diff

44
src/virt/nerdctl/nerdctldocs/compose.md Normal file
View File

@@ -0,0 +1,44 @@
# nerdctl compose
| :zap: Requirement | nerdctl >= 0.8 |
|-------------------|----------------|
## Usage
The `nerdctl compose` CLI is designed to be compatible with `docker-compose`.
```console
$ nerdctl compose up -d
$ nerdctl compose down
```
See the Command Reference in [`../README.md`](../README.md).
## Spec conformance
`nerdctl compose` implements [The Compose Specification](https://github.com/compose-spec/compose-spec),
which was derived from [Docker Compose file version 3 specification](https://docs.docker.com/compose/compose-file/compose-file-v3/).
### Unimplemented YAML fields
- Fields that correspond to unimplemented `docker run` flags, e.g., `services.<SERVICE>.links` (corresponds to `docker run --link`)
- Fields that correspond to unimplemented `docker build` flags, e.g., `services.<SERVICE>.build.extra_hosts` (corresponds to `docker build --add-host`)
- `services.<SERVICE>.credential_spec`
- `services.<SERVICE>.deploy.update_config`
- `services.<SERVICE>.deploy.rollback_config`
- `services.<SERVICE>.deploy.resources.reservations`
- `services.<SERVICE>.deploy.placement`
- `services.<SERVICE>.deploy.endpoint_mode`
- `services.<SERVICE>.healthcheck`
- `services.<SERVICE>.stop_grace_period`
- `services.<SERVICE>.stop_signal`
- `configs.<CONFIG>.external`
- `secrets.<SECRET>.external`
### Incompatibility
#### `services.<SERVICE>.build.context`
- The value must be a local directory path, not a URL.
#### `services.<SERVICE>.secrets`, `services.<SERVICE>.configs`
- `uid`, `gid`: Cannot be specified. The default value is not propagated from `USER` instruction of Dockerfile.
The file owner corresponds to the original file on the host.
- `mode`: Cannot be specified. The file is mounted as read-only, with permission bits that correspond to the original file on the host.

62
src/virt/nerdctl/nerdctldocs/config.md Normal file
View File

@@ -0,0 +1,62 @@
# Configuring nerdctl with `nerdctl.toml`
| :zap: Requirement | nerdctl >= 0.16 |
|-------------------|-----------------|
This document describes the configuration file of nerdctl (`nerdctl.toml`).
This file is unrelated to the configuration file of containerd (`config.toml`) .
## File path
- Rootful mode: `/etc/nerdctl/nerdctl.toml`
- Rootless mode: `~/.config/nerdctl/nerdctl.toml`
The path can be overridden with `$NERDCTL_TOML`.
## Example
```toml
# This is an example of /etc/nerdctl/nerdctl.toml .
# Unrelated to the daemon's /etc/containerd/config.toml .
debug = false
debug_full = false
address = "unix:///run/k3s/containerd/containerd.sock"
namespace = "k8s.io"
snapshotter = "stargz"
cgroup_manager = "cgroupfs"
hosts_dir = ["/etc/containerd/certs.d", "/etc/docker/certs.d"]
experimental = true
```
## Properties
| TOML property | CLI flag | Env var | Description | Availability \*1 |
|---------------------|------------------------------------|---------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
| `debug` | `--debug` | | Debug mode | Since 0.16.0 |
| `debug_full` | `--debug-full` | | Debug mode (with full output) | Since 0.16.0 |
| `address` | `--address`,`--host`,`-a`,`-H` | `$CONTAINERD_ADDRESS` | containerd address | Since 0.16.0 |
| `namespace` | `--namespace`,`-n` | `$CONTAINERD_NAMESPACE` | containerd namespace | Since 0.16.0 |
| `snapshotter` | `--snapshotter`,`--storage-driver` | `$CONTAINERD_SNAPSHOTTER` | containerd snapshotter | Since 0.16.0 |
| `cni_path` | `--cni-path` | `$CNI_PATH` | CNI binary directory | Since 0.16.0 |
| `cni_netconfpath` | `--cni-netconfpath` | `$NETCONFPATH` | CNI config directory | Since 0.16.0 |
| `data_root` | `--data-root` | | Persistent state directory | Since 0.16.0 |
| `cgroup_manager` | `--cgroup-manager` | | cgroup manager | Since 0.16.0 |
| `insecure_registry` | `--insecure-registry` | | Allow insecure registry | Since 0.16.0 |
| `hosts_dir` | `--hosts-dir` | | `certs.d` directory | Since 0.16.0 |
| `experimental` | `--experimental` | `NERDCTL_EXPERIMENTAL` | Enable [experimental features](experimental.md) | Since 0.22.3 |
| `host_gateway_ip` | `--host-gateway-ip` | `NERDCTL_HOST_GATEWAY_IP` | IP address that the special 'host-gateway' string in --add-host resolves to. Defaults to the IP address of the host. It has no effect without setting --add-host | Since 1.3.0 |
| `bridge_ip` | `--bridge-ip` | `NERDCTL_BRIDGE_IP` | IP address for the default nerdctl bridge network, e.g., 10.1.100.1/24 | Since 2.0.1 |
| `kube_hide_dupe` | `--kube-hide-dupe` | | Deduplicate images for Kubernetes with namespace k8s.io, no more redundant <none> ones are displayed | Since 2.0.3 |
The properties are parsed in the following precedence:
1. CLI flag
2. Env var
3. TOML property
4. Built-in default value (Run `nerdctl --help` to see the default values)
\*1: Availability of the TOML properties
## See also
- [`registry.md`](registry.md)
- [`faq.md`](faq.md)
- https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration (`/etc/containerd/config.toml`)

214
src/virt/nerdctl/nerdctldocs/cosign.md Normal file
View File

@@ -0,0 +1,214 @@
# Container Image Sign and Verify with cosign tool
| :zap: Requirement | nerdctl >= 0.15 |
|-------------------|-----------------|
[cosign](https://github.com/sigstore/cosign) is tool that allows you to sign and verify container images with the
public/private key pairs or without them by providing
a [Keyless support](https://github.com/sigstore/cosign/blob/main/KEYLESS.md).
Keyless uses ephemeral keys and certificates, which are signed automatically by
the [fulcio](https://github.com/sigstore/fulcio) root CA. Signatures are stored in
the [rekor](https://github.com/sigstore/rekor) transparency log, which automatically provides an attestation as to when
the signature was created.
Cosign would use prompt to confirm the statement below during `sign`. Nerdctl added `--yes` to Cosign command, which says yes and prevents this prompt.
Using Nerdctl push with signing by Cosign means that users agree the statement.
```
Note that there may be personally identifiable information associated with this signed artifact.
This may include the email address associated with the account with which you authenticate.
This information will be used for signing this artifact and will be stored in public transparency logs and cannot be removed later.
By typing 'y', you attest that you grant (or have permission to grant) and agree to have this information stored permanently in transparency logs.
```
You can enable container signing and verifying features with `push` and `pull` commands of `nerdctl` by using `cosign`
under the hood with make use of flags `--sign` while pushing the container image, and `--verify` while pulling the
container image.
> * Ensure cosign executable in your `$PATH`.
> * You can install cosign by following this page: https://docs.sigstore.dev/cosign/installation
Prepare your environment:
```shell
# Create a sample Dockerfile
$ cat <<EOF | tee Dockerfile.dummy
FROM alpine:latest
CMD [ "echo", "Hello World" ]
EOF
```
> Please do not forget, we won't be validating the base images, which is `alpine:latest` in this case, of the container image that was built on,
> we'll only verify the container image itself once we sign it.
```shell
# Build the image
$ nerdctl build -t devopps/hello-world -f Dockerfile.dummy .
# Generate a key-pair: cosign.key and cosign.pub
$ cosign generate-key-pair
# Export your COSIGN_PASSWORD to prevent CLI prompting
$ export COSIGN_PASSWORD=$COSIGN_PASSWORD
```
Sign the container image while pushing:
```
# Sign the image with Keyless mode
$ nerdctl push --sign=cosign devopps/hello-world
# Sign the image and store the signature in the registry
$ nerdctl push --sign=cosign --cosign-key cosign.key devopps/hello-world
```
Verify the container image while pulling:
> REMINDER: Image won't be pulled if there are no matching signatures in case you passed `--verify` flag.
> REMINDER: For keyless flows to work, you need to set either --cosign-certificate-identity or --cosign-certificate-identity-regexp, and either --cosign-certificate-oidc-issuer or --cosign-certificate-oidc-issuer-regexp. The OIDC issuer expected in a valid Fulcio certificate for --verify=cosign, e.g. https://token.actions.githubusercontent.com or https://oauth2.sigstore.dev/auth.
```shell
# Verify the image with Keyless mode
$ nerdctl pull --verify=cosign --certificate-identity=name@example.com --certificate-oidc-issuer=https://accounts.example.com devopps/hello-world
INFO[0004] cosign:
INFO[0004] cosign: [{"critical":{"identity":...}]
docker.io/devopps/nginx-new:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0910d404e58dd320c3c0c7ea31bf5fbfe7544b26905c5eccaf87c3af7bcf9b88: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:1de1c4fb5122ac8650e349e018fba189c51300cf8800d619e92e595d6ddda40e: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.4 s total: 1.3 Ki (928.0 B/s)
# You can not verify the image if it is not signed
$ nerdctl pull --verify=cosign --cosign-key cosign.pub devopps/hello-world-bad
INFO[0003] cosign: Error: no matching signatures:
INFO[0003] cosign: failed to verify signature
INFO[0003] cosign: main.go:46: error during command execution: no matching signatures:
INFO[0003] cosign: failed to verify signature
```
## Cosign in Compose
> Cosign support in Compose is also experimental and implemented based on Compose's [extension](https://github.com/compose-spec/compose-spec/blob/master/spec.md#extension) capibility.
cosign is supported in `nerdctl compose up|run|push|pull`. You can use cosign in Compose by adding the following fields in your compose yaml. These fields are _per service_, and you can enable only `verify` or only `sign` (or both).
```yaml
# only put cosign related fields under the service you want to sign/verify.
services:
svc0:
build: .
image: ${REGISTRY}/svc0_image # replace with your registry
# `x-nerdctl-verify` and `x-nerdctl-cosign-public-key` are for verify
# required for `nerdctl compose up|run|pull`
x-nerdctl-verify: cosign
x-nerdctl-cosign-public-key: /path/to/cosign.pub
# `x-nerdctl-sign` and `x-nerdctl-cosign-private-key` are for sign
# required for `nerdctl compose push`
x-nerdctl-sign: cosign
x-nerdctl-cosign-private-key: /path/to/cosign.key
ports:
- 8080:80
svc1:
build: .
image: ${REGISTRY}/svc1_image # replace with your registry
ports:
- 8081:80
```
Following the cosign tutorial above, first set up environment and prepare cosign key pair:
```shell
# Generate a key-pair: cosign.key and cosign.pub
$ cosign generate-key-pair
# Export your COSIGN_PASSWORD to prevent CLI prompting
$ export COSIGN_PASSWORD=$COSIGN_PASSWORD
```
We'll use the following `Dockerfile` and `docker-compose.yaml`:
```shell
$ cat Dockerfile
FROM nginx:1.19-alpine
RUN uname -m > /usr/share/nginx/html/index.html
$ cat docker-compose.yml
services:
svc0:
build: .
image: ${REGISTRY}/svc1_image # replace with your registry
x-nerdctl-verify: cosign
x-nerdctl-cosign-public-key: ./cosign.pub
x-nerdctl-sign: cosign
x-nerdctl-cosign-private-key: ./cosign.key
ports:
- 8080:80
svc1:
build: .
image: ${REGISTRY}/svc1_image # replace with your registry
ports:
- 8081:80
```
For keyless mode, the `docker-compose.yaml` will be:
```
$ cat docker-compose.yml
services:
svc0:
build: .
image: ${REGISTRY}/svc1_image # replace with your registry
x-nerdctl-verify: cosign
x-nerdctl-sign: cosign
x-nerdctl-cosign-certificate-identity: name@example.com # or x-nerdctl-cosign-certificate-identity-regexp
x-nerdctl-cosign-certificate-oidc-issuer: https://accounts.example.com # or x-nerdctl-cosign-certificate-oidc-issuer-regexp
ports:
- 8080:80
svc1:
build: .
image: ${REGISTRY}/svc1_image # replace with your registry
ports:
- 8081:80
```
> The `env "COSIGN_PASSWORD="$COSIGN_PASSWORD""` part in the below commands is a walkaround to use rootful nerdctl and make the env variable visible to root (in sudo). You don't need this part if (1) you're using rootless, or (2) your `COSIGN_PASSWORD` is visible in root.
First let's `build` and `push` the two services:
```shell
$ sudo nerdctl compose build
INFO[0000] Building image xxxxx/svc0_image
...
INFO[0000] Building image xxxxx/svc1_image
[+] Building 0.2s (6/6) FINISHED
$ sudo env "COSIGN_PASSWORD="$COSIGN_PASSWORD"" nerdctl compose --experimental=true push
INFO[0000] Pushing image xxxxx/svc1_image
...
INFO[0000] Pushing image xxxxx/svc0_image
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:4329abc3143b1545835de17e1302c8313a9417798b836022f4c8c8dc8b10a3e9)
INFO[0000] cosign: WARNING: Image reference xxxxx/svc0_image uses a tag, not a digest, to identify the image to sign.
INFO[0000] cosign:
INFO[0000] cosign: This can lead you to sign a different image than the intended one. Please use a
INFO[0000] cosign: digest (example.com/ubuntu@sha256:abc123...) rather than tag
INFO[0000] cosign: (example.com/ubuntu:latest) for the input to cosign. The ability to refer to
INFO[0000] cosign: images by tag will be removed in a future release.
INFO[0000] cosign: Pushing signature to: xxxxx/svc0_image
```
Then we can `pull` and `up` services (`run` is similar to up):
```shell
# ensure built images are removed and pull is performed.
$ sudo nerdctl compose down
$ sudo env "COSIGN_PASSWORD="$COSIGN_PASSWORD"" nerdctl compose --experimental=true pull
$ sudo env "COSIGN_PASSWORD="$COSIGN_PASSWORD"" nerdctl compose --experimental=true up
$ sudo env "COSIGN_PASSWORD="$COSIGN_PASSWORD"" nerdctl compose --experimental=true run svc0 -- echo "hello"
# clean up compose resources.
$ sudo nerdctl compose down
```
Check your logs to confirm that svc0 is verified by cosign (have cosign logs) and svc1 is not. You can also change the public key in `docker-compose.yaml` to a random value to see verify failure will stop the container being `pull|up|run`.

89
src/virt/nerdctl/nerdctldocs/cvmfs.md Normal file
View File

@@ -0,0 +1,89 @@
# Lazy-pulling using CernVM-FS Snapshotter
CernVM-FS Snapshotter is a containerd snapshotter plugin. It is a specialized component responsible for assembling
all the layers of container images into a stacked file system that containerd can use. The snapshotter takes as input the list
of required layers and outputs a directory containing the final file system. It is also responsible to clean up the output
directory when containers using it are stopped.
See the official [documentation](https://cvmfs.readthedocs.io/en/latest/cpt-containers.html#how-to-use-the-cernvm-fs-snapshotter) to learn further information.
## Prerequisites
- Install containerd remote snapshotter plugin (`cvmfs-snapshotter`) from [here](https://github.com/cvmfs/cvmfs/tree/devel/snapshotter).
- Add the following to `/etc/containerd/config.toml`:
```toml
# Ask containerd to use this particular snapshotter
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "cvmfs-snapshotter"
disable_snapshot_annotations = false
# Set the communication endpoint between containerd and the snapshotter
[proxy_plugins]
[proxy_plugins.cvmfs]
type = "snapshot"
address = "/run/containerd-cvmfs-grpc/containerd-cvmfs-grpc.sock"
```
- The default CernVM-FS repository hosting the flat root filesystems of the container images is `unpacked.cern.ch`.
The container images are unpacked into the CernVM-FS repository by the [DUCC](https://cvmfs.readthedocs.io/en/latest/cpt-ducc.html)
(Daemon that Unpacks Container Images into CernVM-FS) tool.
You can change the repository adding the following line to `/etc/containerd-cvmfs-grpc/config.toml`:
```toml
repository = "myrepo.mydomain"
```
- Launch `containerd` and `cvmfs-snapshotter`:
```console
$ systemctl start containerd cvmfs-snapshotter
```
## Enable CernVM-FS Snapshotter for `nerdctl run` and `nerdctl pull`
| :zap: Requirement | nerdctl >= 1.6.3 |
| ----------------- | ---------------- |
- Run `nerdctl` with `--snapshotter cvmfs-snapshotter` as in the example below:
```console
$ nerdctl run -it --rm --snapshotter cvmfs-snapshotter clelange/cms-higgs-4l-full:latest
```
- You can also only pull the image with CernVM-FS Snapshotter without running the container:
```console
$ nerdctl pull --snapshotter cvmfs-snapshotter clelange/cms-higgs-4l-full:latest
```
The speedup for pulling this 9 GB (4.3 GB compressed) image is shown below:
- #### with the snapshotter:
```console
$ nerdctl --snapshotter cvmfs-snapshotter pull clelange/cms-higgs-4l-full:latest
docker.io/clelange/cms-higgs-4l-full:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:b8acbe80629dd28d213c03cf1ffd3d46d39e573f54215a281fabce7494b3d546: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:89ef54b6c4fbbedeeeb29b1df2b9916b6d157c87cf1878ea882bff86a3093b5c: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 4.7 s total: 19.8 K (4.2 KiB/s)
$ nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
clelange/cms-higgs-4l-full latest b8acbe80629d 20 seconds ago linux/amd64 0.0 B 4.3 GiB
```
- #### without the snapshotter:
```console
$ nerdctl pull clelange/cms-higgs-4l-full:latest
docker.io/clelange/cms-higgs-4l-full:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:b8acbe80629dd28d213c03cf1ffd3d46d39e573f54215a281fabce7494b3d546: exists |++++++++++++++++++++++++++++++++++++++|
config-sha256:89ef54b6c4fbbedeeeb29b1df2b9916b6d157c87cf1878ea882bff86a3093b5c: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e8114d4b0d10b33aaaa4fbc3c6da22bbbcf6f0ef0291170837e7c8092b73840a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a3eda0944a81e87c7a44b117b1c2e707bc8d18e9b7b478e21698c11ce3e8b819: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8f3160776e8e8736ea9e3f6c870d14cd104143824bbcabe78697315daca0b9ad: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:22a5c05baa9db0aa7bba56ffdb2dd21246b9cf3ce938fc6d7bf20e92a067060e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bfcf9d498f92b72426c9d5b73663504d87249d6783c6b58d71fbafc275349ab9: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0563e1549926b9c8beac62407bc6a420fa35bcf6f9844e5d8beeb9165325a872: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6fff5fd7fb4eeb79a1399d9508614a84191d05e53f094832062d689245599640: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:25c39bfa66e1157415236703abc512d06cc1db31bd00fe8c3030c6d6d249dc4e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3cc0a0eb55eb3fb7ef0760c6bf1e567dfc56933ba5f11b5415f89228af751b72: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a8850244786303e508b94bb31c8569310765e678c9c73bf1199310729209b803: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:32cdf5fc12485ac061347eb8b5c3b4a28505ce8564a7f3f83ac4241f03911176: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 181.8s total: 4.3 Gi (24.2 MiB/s)
$ nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
clelange/cms-higgs-4l-full latest b8acbe80629d 4 minutes ago linux/amd64 9.0 GiB 4.3 GiB
```

73
src/virt/nerdctl/nerdctldocs/dir.md Normal file
View File

@@ -0,0 +1,73 @@
# nerdctl directory layout
## Config
**Default**: `/etc/nerdctl/nerdctl.toml` (rootful), `~/.config/nerdctl/nerdctl.toml` (rootless)
The configuration file of nerdctl. See [`config.md`](./config.md).
Can be overridden with environment variable `$NERDCTL_TOML`.
This file is unrelated to the daemon config file `/etc/containerd/config.toml`.
## Data
### `<DATAROOT>`
**Default**: `/var/lib/nerdctl` (rootful), `~/.local/share/nerdctl` (rootless)
Can be overridden with `nerdctl --data-root=<DATAROOT>` flag.
The directory is solely managed by nerdctl, not by containerd.
The directory has nothing to do with containerd data root `/var/lib/containerd`.
### `<DATAROOT>/<ADDRHASH>`
e.g. `/var/lib/nerdctl/1935db59`
`1935db9` is from `$(echo -n "/run/containerd/containerd.sock" | sha256sum | cut -c1-8)`
This directory is also called "data store" in the implementation.
### `<DATAROOT>/<ADDRHASH>/containers/<NAMESPACE>/<CID>`
e.g. `/var/lib/nerdctl/1935db59/containers/default/c4ed811cc361d26faffdee8d696ddbc45a9d93c571b5b3c54d3da01cb29caeb1`
Files:
- `resolv.conf`: mounted to the container as `/etc/resolv.conf`
- `hostname`: mounted to the container as `/etc/hostname`
- `log-config.json`: used for storing the `--log-opts` map of `nerdctl run`
- `<CID>-json.log`: used by `nerdctl logs`
- `oci-hook.*.log`: logs of the OCI hook
- `lifecycle.json`: used to store stateful information about the container that can only be retrieved through OCI hooks
### `<DATAROOT>/<ADDRHASH>/names/<NAMESPACE>`
e.g. `/var/lib/nerdctl/1935db59/names/default`
Files:
- `<NAME>`: contains the container ID (CID). Represents that the name is taken by that container.
Files must be operated with a `LOCK_EX` lock against the `<DATAROOT>/<ADDRHASH>/names/<NAMESPACE>` directory.
### `<DATAROOT>/<ADDRHASH>/etchosts/<NAMESPACE>/<CID>`
e.g. `/var/lib/nerdctl/1935db59/etchosts/default/c4ed811cc361d26faffdee8d696ddbc45a9d93c571b5b3c54d3da01cb29caeb1`
Files:
- `hosts`: mounted to the container as `/etc/hosts`
- `meta.json`: metadata
Files must be operated with a `LOCK_EX` lock against the `<DATAROOT>/<ADDRHASH>/etchosts` directory.
### `<DATAROOT>/<ADDRHASH>/volumes/<NAMESPACE>/<VOLNAME>/_data`
e.g. `/var/lib/nerdctl/1935db59/volumes/default/foo/_data`
Data volume
## CNI
### `<NETCONFPATH>`
**Default**: `/etc/cni/net.d` (rootful), `~/.config/cni/net.d` (rootless)
Can be overridden with `nerdctl --cni-netconfpath=<NETCONFPATH>` flag and environment variable `$NETCONFPATH`.
At the top-level of <NETCONFPATH>, network (files) are shared accross all namespaces.
Sub-folders inside <NETCONFPATH> are only available to the namespace bearing the same name,
and its networks definitions are private.
Files:
- `nerdctl-<NWNAME>.conflist`: CNI conf list created by nerdctl

85
src/virt/nerdctl/nerdctldocs/gpu.md Normal file
View File

@@ -0,0 +1,85 @@
# Using GPUs inside containers
| :zap: Requirement | nerdctl >= 0.9 |
|-------------------|----------------|
nerdctl provides docker-compatible NVIDIA GPU support.
## Prerequisites
- NVIDIA Drivers
- Same requirement as when you use GPUs on Docker. For details, please refer to [the doc by NVIDIA](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#pre-requisites).
- `nvidia-container-cli`
- containerd relies on this CLI for setting up GPUs inside container. You can install this via [`libnvidia-container` package](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/arch-overview.html#libnvidia-container).
## Options for `nerdctl run --gpus`
`nerdctl run --gpus` is compatible to [`docker run --gpus`](https://docs.docker.com/engine/reference/commandline/run/#access-an-nvidia-gpu).
You can specify number of GPUs to use via `--gpus` option.
The following example exposes all available GPUs.
```
nerdctl run -it --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu20.04 nvidia-smi
```
You can also pass detailed configuration to `--gpus` option as a list of key-value pairs. The following options are provided.
- `count`: number of GPUs to use. `all` exposes all available GPUs.
- `device`: IDs of GPUs to use. UUID or numbers of GPUs can be specified.
- `capabilities`: [Driver capabilities](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#driver-capabilities). If unset, use default driver `utility`, `compute`.
The following example exposes a specific GPU to the container.
```
nerdctl run -it --rm --gpus '"capabilities=utility,compute",device=GPU-3a23c669-1f69-c64e-cf85-44e9b07e7a2a' nvidia/cuda:12.3.1-base-ubuntu20.04 nvidia-smi
```
## Fields for `nerdctl compose`
`nerdctl compose` also supports GPUs following [compose-spec](https://github.com/compose-spec/compose-spec/blob/master/deploy.md#devices).
You can use GPUs on compose when you specify some of the following `capabilities` in `services.demo.deploy.resources.reservations.devices`.
- `gpu`
- `nvidia`
- all allowed capabilities for `nerdctl run --gpus`
Available fields are the same as `nerdctl run --gpus`.
The following exposes all available GPUs to the container.
```
version: "3.8"
services:
demo:
image: nvidia/cuda:12.3.1-base-ubuntu20.04
command: nvidia-smi
deploy:
resources:
reservations:
devices:
- capabilities: ["utility"]
count: all
```
## Trouble Shooting
### `nerdctl run --gpus` fails when using the Nvidia gpu-operator
If the Nvidia driver is installed by the [gpu-operator](https://github.com/NVIDIA/gpu-operator).The `nerdctl run` will fail with the error message `(FATA[0000] exec: "nvidia-container-cli": executable file not found in $PATH)`.
So, the `nvidia-container-cli` needs to be added to the PATH environment variable.
You can do this by adding the following line to your $HOME/.profile or /etc/profile (for a system-wide installation):
```
export PATH=$PATH:/usr/local/nvidia/toolkit
```
The shared libraries also need to be added to the system.
```
echo "/run/nvidia/driver/usr/lib/x86_64-linux-gnu" > /etc/ld.so.conf.d/nvidia.conf
ldconfig
```
And then, the `nerdctl run --gpus` can run successfully.

292
src/virt/nerdctl/nerdctldocs/ipfs.md Normal file
View File

@@ -0,0 +1,292 @@
# Distribute Container Images on IPFS (Experimental)
| :zap: Requirement | nerdctl >= 0.14 |
|-------------------|-----------------|
You can distribute container images without registries, using IPFS.
IPFS support is completely optional. Your host is NOT connected to any P2P network, unless you opt in to [install and run IPFS daemon](https://docs.ipfs.io/install/).
## Prerequisites
### ipfs daemon
Make sure an IPFS daemon such as [Kubo](https://github.com/ipfs/kubo) (former go-ipfs) is running on your host.
For example, you can run Kubo using the following command.
```
ipfs daemon
```
In rootless mode, you need to install ipfs daemon using `containerd-rootless-setuptool.sh`.
```
containerd-rootless-setuptool.sh -- install-ipfs --init
```
> NOTE: correctly set IPFS_PATH as described in the output of the above command.
:information_source: If you want to expose some ports of ipfs daemon (e.g. 4001), you can install rootless containerd using `containerd-rootless-setuptool.sh install` with `CONTAINERD_ROOTLESS_ROOTLESSKIT_FLAGS="--publish=0.0.0.0:4001:4001/tcp"` environment variable.
:information_source: If you don't want IPFS to communicate with nodes on the internet, you can run IPFS daemon in offline mode using `--offline` flag or you can create a private IPFS network as described [here](https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md#appendix-1-creating-ipfs-private-network).
:information_source: Instead of locally launching IPFS daemon, you can specify the address of the IPFS API using `--ipfs-address` flag.
## IPFS-enabled image and OCI Compatibility
Image distribution on IPFS is achieved by OCI-compatible *IPFS-enabled image format*.
nerdctl automatically converts an image to IPFS-enabled when necessary.
For example, when nerdctl pushes an image to IPFS, if that image isn't an IPFS-enabled one, it converts that image to the IPFS-enabled one.
Please see [the doc in stargz-snapshotter project](https://github.com/containerd/stargz-snapshotter/blob/v0.10.0/docs/ipfs.md) for details about IPFS-enabled image format.
## Using nerdctl with IPFS
nerdctl supports an image name prefix `ipfs://` to handle images on IPFS.
### `nerdctl push ipfs://<image-name>`
For `nerdctl push`, you can specify `ipfs://` prefix for arbitrary image names stored in containerd.
When this prefix is specified, nerdctl pushes that image to IPFS.
```console
> nerdctl push ipfs://ubuntu:20.04
INFO[0000] pushing image "ubuntu:20.04" to IPFS
INFO[0000] ensuring image contents
bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
```
At last line of the output, the IPFS CID of the pushed image is printed.
You can use this CID to pull this image from IPFS.
You can also specify `--estargz` option to enable [eStargz-based lazy pulling](https://github.com/containerd/stargz-snapshotter/blob/v0.10.0/docs/ipfs.md) on IPFS.
Please see the later section for details.
```console
> nerdctl push --estargz ipfs://fedora:36
INFO[0000] pushing image "fedora:36" to IPFS
INFO[0000] ensuring image contents
INFO[0011] converted "application/vnd.docker.image.rootfs.diff.tar.gzip" to sha256:cd4be969f12ef45dee7270f3643f796364045edf94cfa9ef6744d91d5cdf2208
bafkreibp2ncujcia663uum25ustwvmyoguxqyzjnxnlhebhsgk2zowscye
```
### `nerdctl pull ipfs://<CID>` and `nerdctl run ipfs://<CID>`
You can pull an image from IPFS by specifying `ipfs://<CID>` where `CID` is the CID of the image.
```console
> nerdctl pull ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:28bfa1fc6d491d3bee91bab451cab29c747e72917efacb0adc4e73faffe1f51c: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f6eed19a2880f1000be1d46fb5d114d094a59e350f9d025580f7297c8d9527d5: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.2 s total: 27.2 M (22.7 MiB/s)
```
`nerdctl run` also supports the same image name syntax.
When specified, this command pulls the image from IPFS.
```console
> nerdctl run --rm -it ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze echo hello
hello
```
You can also push that image to the container registry.
```
nerdctl tag ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze ghcr.io/ktock/ubuntu:20.04-ipfs
nerdctl push ghcr.io/ktock/ubuntu:20.04-ipfs
```
The pushed image can run on other (IPFS-agnostic) runtimes.
```console
> docker run --rm -it ghcr.io/ktock/ubuntu:20.04-ipfs echo hello
hello
```
:information_source: Note that though the IPFS-enabled image is OCI compatible, some runtimes including [containerd](https://github.com/containerd/containerd/pull/6221) and [podman](https://github.com/containers/image/pull/1403) had bugs and failed to pull that image. Containerd fixed this since v1.5.8, podman fixed this since commit [`b55fb86c28b7d743cf59701332cd78d4294c7c54`](https://github.com/containers/image/commit/b55fb86c28b7d743cf59701332cd78d4294c7c54).
### `nerdctl build` and `localhost:5050/ipfs/<CID>` image reference
You can build images using base images on IPFS.
BuildKit >= v0.9.3 is needed.
In Dockerfile, instead of `ipfs://` prefix, you need to use the following image reference to point to an image on IPFS.
```
localhost:5050/ipfs/<CID>
```
Here, `CID` is the IPFS CID of the image.
:information_source: In the future version of nerdctl and BuildKit, `ipfs://` prefix should be supported in Dockerfile.
Using this image reference, you can build an image on IPFS.
```dockerfile
FROM localhost:5050/ipfs/bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
RUN echo hello > /hello
```
Make sure that `nerdctl ipfs registry serve` is running.
This allows `nerdctl build` to pull images from IPFS.
```
$ nerdctl ipfs registry serve &
```
Then you can build this Dockerfile using `nerdctl build`.
```console
> nerdctl build -t hello .
[+] Building 5.3s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 146B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for localhost:5050/ipfs/bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze:latest 0.1s
=> [1/2] FROM localhost:5050/ipfs/bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze@sha256:28bfa1fc6d491d3bee91bab451cab29c747e72917e 3.8s
=> => resolve localhost:5050/ipfs/bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze@sha256:28bfa1fc6d491d3bee91bab451cab29c747e72917e 0.0s
=> => sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54 28.57MB / 28.57MB 2.1s
=> => extracting sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54 1.7s
=> [2/2] RUN echo hello > /hello 0.6s
=> exporting to oci image format 0.6s
=> => exporting layers 0.1s
=> => exporting manifest sha256:b96d490d134221ab121af91a42b13195dd8c5bf941012d7bfe07eabcf5259eda 0.0s
=> => exporting config sha256:bd706574eab19009585b98826b06e63cf6eacf8d7193504dae75caa760332ca2 0.0s
=> => sending tarball 0.5s
unpacking docker.io/library/hello:latest (sha256:b96d490d134221ab121af91a42b13195dd8c5bf941012d7bfe07eabcf5259eda)...done
> nerdctl run --rm -it hello cat /hello
hello
```
> NOTE: `--ipfs` flag has been removed since v1.2.0. You need to launch the localhost registry by yourself using `nerdctl ipfs registry serve`.
#### Details about `localhost:5050/ipfs/<CID>` and `nerdctl ipfs registry`
As of now, BuildKit doesn't support `ipfs://` prefix so nerdctl achieves builds on IPFS by having a read-only local registry backed by IPFS.
This registry converts registry API requests to IPFS operations.
So IPFS-agnostic tools can pull images from IPFS via this registry.
This registry is provided as a subcommand `nerdctl ipfs registry`.
This command starts the registry backed by the IPFS repo of the current `$IPFS_PATH`
By default, nerdctl exposes the registry at `localhost:5050` (configurable via flags).
<details>
<summary>Creating systemd unit file for `nerdctl ipfs registry`</summary>
Optionally you can create systemd unit file of `nerdctl ipfs registry serve`.
An example systemd unit file for `nerdctl ipfs registry serve` can be the following.
`nerdctl ipfs registry serve` is aware of environment variables for configuring the behaviour (e.g. listening port) so you can use `EnvironmentFile` for configuring it.
```
[Unit]
Description=nerdctl ipfs registry serve
[Service]
EnvironmentFile-=/run/nerdctl-ipfs-registry-serve/env
ExecStart=nerdctl ipfs registry serve
[Install]
WantedBy=default.target
```
</details>
The following example starts the registry on `localhost:5555` instead of `localhost:5050`.
```
nerdctl ipfs registry serve --listen-registry=localhost:5555
```
> NOTE: You'll also need to restart the registry when you change `$IPFS_PATH` to use.
> NOTE: `nerdctl ipfs registry [up|down]` has been removed since v1.2.0. You need to launch the localhost registry using `nerdctl ipfs registry serve` instead.
### Compose on IPFS
`nerdctl compose` supports same image name syntax to pull images from IPFS.
```yaml
version: "3.8"
services:
ubuntu:
image: ipfs://bafkreicq4dg6nkef5ju422ptedcwfz6kcvpvvhuqeykfrwq5krazf3muze
command: echo hello
```
When you build images using base images on IPFS, you can use `localhost:5050/ipfs/<CID>` image reference in Dockerfile as mentioned above.
```
nerdctl compose up --build
```
```
nerdctl compose build
```
> NOTE: `--ipfs` flag has been removed since v1.2.0. You need to launch the localhost registry by yourself using `nerdctl ipfs registry serve`.
### Encryption
You can distribute [encrypted images](./ocicrypt.md) on IPFS using OCIcrypt.
Please see [`/docs/ocicrypt.md`](./ocicrypt.md) for details about how to encrypt and decrypt an image.
Same as normal images, the encrypted image can be pushed to IPFS using `ipfs://` prefix.
```console
> nerdctl image encrypt --recipient=jwe:mypubkey.pem ubuntu:20.04 ubuntu:20.04-encrypted
sha256:a5c57411f3d11bb058b584934def0710c6c5b5a4a2d7e9b78f5480ecfc450740
> nerdctl push ipfs://ubuntu:20.04-encrypted
INFO[0000] pushing image "ubuntu:20.04-encrypted" to IPFS
INFO[0000] ensuring image contents
bafkreifajsysbvhtgd7fdgrfesszexdq6v5zbj5y2jnjfwxdjyqws2s3s4
```
You can pull the encrypted image from IPFS using `ipfs://` prefix and can decrypt it in the same way as described in [`/docs/ocicrypt.md`](./ocicrypt.md).
```console
> nerdctl pull --unpack=false ipfs://bafkreifajsysbvhtgd7fdgrfesszexdq6v5zbj5y2jnjfwxdjyqws2s3s4
bafkreifajsysbvhtgd7fdgrfesszexdq6v5zbj5y2jnjfwxdjyqws2s3s4: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:73334fee83139d1d8dbf488b28ad100767c38428b2a62504c758905c475c1d6c: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:8855ae825902045ea2b27940634673ba410b61885f91b9f038f6b3303f48727c: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e74a9a7749e808e4ad1e90d5a81ce3146ce270de0fbdf22429cd465df8f10a13: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.3 s total: 22.0 M (73.2 MiB/s)
> nerdctl image decrypt --key=mykey.pem ipfs://bafkreifajsysbvhtgd7fdgrfesszexdq6v5zbj5y2jnjfwxdjyqws2s3s4 ubuntu:20.04-decrypted
sha256:b0ccaddb7e7e4e702420de126468eab263eb0f3c25abf0b957ce8adcd1e82105
> nerdctl run --rm -it ubuntu:20.04-decrypted echo hello
hello
```
## Running containers on IPFS with eStargz-based lazy pulling
nerdctl supports running eStargz images on IPFS with lazy pulling using Stargz Snapshotter.
In this configuration, Stargz Snapshotter mounts the eStargz image from IPFS to the container's rootfs using FUSE with lazy pulling support.
Thus the container can startup without waiting for the entire image contents to be locally available.
You can see faster container cold-start.
To use this feature, you need to enable Stargz Snapshotter following [`/docs/stargz.md`](./stargz.md).
You also need to add the following configuration to `config.toml` of Stargz Snapshotter (typically located at `/etc/containerd-stargz-grpc/config.toml`).
```toml
ipfs = true
```
You can push an arbitrary image to IPFS with converting it to eStargz using `--estargz` option.
```
nerdctl push --estargz ipfs://fedora:36
```
You can pull and run that eStargz image with lazy pulling.
```
nerdctl run --rm -it ipfs://bafkreibp2ncujcia663uum25ustwvmyoguxqyzjnxnlhebhsgk2zowscye echo hello
```
- See [the doc in stargz-snapshotter project](https://github.com/containerd/stargz-snapshotter/blob/v0.10.0/docs/ipfs.md) for details about lazy pulling on IPFS.
- See [`/docs/stargz.md`](./stargz.md) for details about the configuration of nerdctl for Stargz Snapshotter.

72
src/virt/nerdctl/nerdctldocs/multi-platform.md Normal file
View File

@@ -0,0 +1,72 @@
# Multi-platform
| :zap: Requirement | nerdctl >= 0.13 |
|-------------------|-----------------|
nerdctl can execute non-native container images using QEMU.
e.g., ARM on Intel, and vice versa.
## Preparation: Register QEMU to `/proc/sys/fs/binfmt_misc`
```console
$ sudo systemctl start containerd
$ sudo nerdctl run --privileged --rm tonistiigi/binfmt:master --install all
$ ls -1 /proc/sys/fs/binfmt_misc/qemu*
/proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-arm
/proc/sys/fs/binfmt_misc/qemu-mips64
/proc/sys/fs/binfmt_misc/qemu-mips64el
/proc/sys/fs/binfmt_misc/qemu-ppc64le
/proc/sys/fs/binfmt_misc/qemu-riscv64
/proc/sys/fs/binfmt_misc/qemu-s390x
```
The `tonistiigi/binfmt` container must be executed with `--privileged`, and with rootful mode (`sudo`).
This container is not a daemon, and exits immediately after registering QEMU to `/proc/sys/fs/binfmt_misc`.
Run `ls -1 /proc/sys/fs/binfmt_misc/qemu*` to confirm registration.
See also https://github.com/tonistiigi/binfmt
## Usage
### Pull & Run
```console
$ nerdctl pull --platform=arm64,s390x alpine
$ nerdctl run --rm --platform=arm64 alpine uname -a
Linux e6227935cf12 5.13.0-19-generic #19-Ubuntu SMP Thu Oct 7 21:58:00 UTC 2021 aarch64 Linux
$ nerdctl run --rm --platform=s390x alpine uname -a
Linux b39da08fbdbf 5.13.0-19-generic #19-Ubuntu SMP Thu Oct 7 21:58:00 UTC 2021 s390x Linux
```
### Build & Push
```console
$ nerdctl build --platform=amd64,arm64 --output type=image,name=example.com/foo:latest,push=true .
```
Or
```console
$ nerdctl build --platform=amd64,arm64 -t example.com/foo:latest .
$ nerdctl push --all-platforms example.com/foo:latest
```
### Compose
See [`../examples/compose-multi-platform`](../examples/compose-multi-platform)
## macOS + Lima
As of 2025-03-01, qemu seems to be broken in most Apple-silicon setups.
This might be due to qemu handling of host vs. guest page sizes
(unconfirmed, see https://github.com/containerd/nerdctl/issues/3948 for more information).
It should also be noted that Linux 6.11 introduced a change to the VDSO (on ARM)
that does break Rosetta.
The take-away here is that presumably your only shot at running non-native binaries
on Apple-silicon is to use an older kernel for your guest (<6.11), typically as shipped by Debian stable,
and also to use VZ+Rosetta and not qemu (eg: `limactl create --vm-type=vz --rosetta`).

81
src/virt/nerdctl/nerdctldocs/notation.md Normal file
View File

@@ -0,0 +1,81 @@
# Container Image Sign and Verify with notation tool
| :zap: Requirement | nerdctl >= 1.3.0 |
|-------------------|------------------|
[notation](https://github.com/notaryproject/notation) is a project to add signatures as standard items in the registry ecosystem, and to build a set of simple tooling for signing and verifying these signatures.
You can enable container signing and verifying features with `push` and `pull` commands of `nerdctl` by using `notation`
under the hood with make use of flags `--sign` while pushing the container image, and `--verify` while pulling the
container image.
* Ensure notation executable in your `$PATH`.
* You can install notation by following this page: https://notaryproject.dev/docs/user-guides/installation/cli/
* Notation follows the RC of OCI spec v1.1.0. Follow the [instruction](https://notaryproject.dev/docs/quickstart/#create-an-oci-compatible-registry) to set up the local registry with the compliance for testing purpose.
Prepare your environment:
```shell
# Create a sample Dockerfile
$ cat <<EOF | tee Dockerfile.dummy
FROM alpine:latest
CMD [ "echo", "Hello World" ]
EOF
```
> Please do not forget, we won't be validating the base images, which is `alpine:latest` in this case, of the container image that was built on,
> we'll only verify the container image itself once we sign it.
```shell
# Build the image
$ nerdctl build -t localhost:5000/my-test -f Dockerfile.dummy .
# Generate a key-pair in notation's key store and trust store
$ notation cert generate-test --default "test"
# Confirm the signing key is correctly configured. Key name with a * prefix is the default key.
$ notation key ls
# Confirm the certificate is stored in the trust store.
$ notation cert ls
```
Sign the container image while pushing:
```
# Sign the image and store the signature in the registry
$ nerdctl push --sign=notation --notation-key-name test localhost:5000/my-test
```
Verify the container image while pulling:
> REMINDER: Image won't be pulled if there are no matching signatures with the cert in the [trust policy](https://github.com/notaryproject/specifications/blob/main/specs/trust-store-trust-policy.md#trust-policy) in case you passed `--verify` flag.
```shell
# Create `trustpolicy.json` under $XDG_CONFIG_HOME/notation (XDG_CONFIG_HOME is ~/.config below)
cat <<EOF | tee ~/.config/notation/trustpolicy.json
{
"version": "1.0",
"trustPolicies": [
{
"name": "test-images",
"registryScopes": [ "*" ],
"signatureVerification": {
"level" : "strict"
},
"trustStores": [ "ca:test" ],
"trustedIdentities": [
"*"
]
}
]
}
EOF
# Verify the image
$ nerdctl pull --verify=notation localhost:5000/my-test
# You can not verify the image if it is not signed by the cert in the trust policy
$ nerdctl pull --verify=notation localhost:5000/my-test-bad
```

37
src/virt/nerdctl/nerdctldocs/nydus.md Normal file
View File

@@ -0,0 +1,37 @@
# Lazy-pulling using Nydus Snapshotter
| :zap: Requirement | nerdctl >= 0.22 |
| ----------------- | --------------- |
Nydus snapshotter is a remote snapshotter plugin of containerd for [Nydus](https://github.com/dragonflyoss/image-service) image service which implements a chunk-based content-addressable filesystem that improves the current OCI image specification, in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity with several runtime backends: FUSE, virtiofs and in-kernel EROFS (Linux kernel 5.19+).
## Enable lazy-pulling for `nerdctl run`
- Install containerd remote snapshotter plugin (`containerd-nydus-grpc`) from https://github.com/containerd/nydus-snapshotter
- Add the following to `/etc/containerd/config.toml`:
```toml
[proxy_plugins]
[proxy_plugins.nydus]
type = "snapshot"
address = "/run/containerd-nydus-grpc/containerd-nydus-grpc.sock"
```
- Launch `containerd` and `containerd-nydus-grpc`
- Run `nerdctl` with `--snapshotter=nydus`
```console
# nerdctl --snapshotter=nydus run -it --rm ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5
```
For the list of pre-converted Nydus images, see https://github.com/orgs/dragonflyoss/packages?page=1&repo_name=image-service
## Build Nydus image using `nerdctl image convert`
Nerdctl supports to convert an OCI image or docker format v2 image to Nydus image by using the `nerdctl image convert` command.
Before the conversion, you should have the `nydus-image` binary installed, which is contained in the ["nydus static package"](https://github.com/dragonflyoss/image-service/releases). You can run the command like `nerdctl image convert --nydus --oci --nydus-builder-path <the_path_of_nydus_image_binary> <source_image> <target_image>` to convert the `<source_image>` to a Nydus image whose tag is `<target_image>`.
By now, the converted Nydus image cannot be run directly. It shoud be unpacked to nydus snapshotter before `nerdctl run`, which is a part of the processing flow of `nerdctl image pull`. So you need to push the converted image to a registry after the conversion and use `nerdctl --snapshotter nydus image pull` to unpack it to the nydus snapshotter before running the image.
Optionally, you can use the nydusify conversion tool to check if the format of the converted Nydus image is valid. For more details about the Nydus image validation and how to build Nydus image, please refer to [nydusify](https://github.com/dragonflyoss/image-service/blob/master/docs/nydusify.md) and [acceld](https://github.com/goharbor/acceleration-service).

90
src/virt/nerdctl/nerdctldocs/ocicrypt.md Normal file
View File

@@ -0,0 +1,90 @@
# OCIcrypt
| :zap: Requirement | nerdctl >= 0.7 |
|-------------------|----------------|
nerdctl supports encryption and decryption using [OCIcrypt](https://github.com/containers/ocicrypt)
(aka [imgcrypt](https://github.com/containerd/imgcrypt) for containerd).
## JWE mode
### Encryption
Use `openssl` to create a private key (`mykey.pem`) and the corresponding public key (`mypubkey.pem`):
```bash
openssl genrsa -out mykey.pem
openssl rsa -in mykey.pem -pubout -out mypubkey.pem
```
Use `nerdctl image encrypt` to create an encrypted image:
```bash
nerdctl image encrypt --recipient=jwe:mypubkey.pem --platform=linux/amd64,linux/arm64 foo example.com/foo:encrypted
nerdctl push example.com/foo:encrypted
```
:warning: CAUTION: This command only encrypts image layers, but does NOT encrypt [container configuration such as `Env` and `Cmd`](https://github.com/opencontainers/image-spec/blob/v1.0.1/config.md#example).
To see non-encrypted information, run `nerdctl image inspect --mode=native --platform=PLATFORM example.com/foo:encrypted` .
### Decryption
#### Configuration
Put the private key files to `/etc/containerd/ocicrypt/keys` (for rootless `~/.config/containerd/ocicrypt/keys`).
<details>
<summary>Extra step for containerd 1.4 and older</summary>
<p>
containerd 1.4 and older requires adding the following configuration to `/etc/containerd/config.toml`
(for rootless `~/.config/containerd/config.toml`):
```toml
version = 2
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
path = "ctd-decoder"
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar"
path = "ctd-decoder"
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
# NOTE: On rootless, ~/.config/containerd is mounted as /etc/containerd in the namespace.
```
</p>
</details>
#### Running nerdctl
No flag is needed for running encrypted images with `nerdctl run`, as long as the private key is stored
in `/etc/containerd/ocicrypt/keys` (for rootless `~/.config/containerd/ocicrypt/keys`).
Just run `nerdctl run example.com/encrypted-image`.
To decrypt an image without running a container, use `nerdctl image decrypt` command:
```bash
nerdctl pull --unpack=false example.com/foo:encrypted
nerdctl image decrypt --key=mykey.pem example.com/foo:encrypted foo:decrypted
```
## PGP (GPG) mode
(Undocumented yet)
## PKCS7 mode
(Undocumented yet)
## PKCS11 mode
(Undocumented yet)
## More information
- https://github.com/containerd/imgcrypt (High-level library for containerd, using `containers/ocicrypt`)
- https://github.com/containers/ocicrypt (Low-level library, used by `containerd/imgcrypt`)
- https://github.com/opencontainers/image-spec/pull/775 (Proposal for OCI Image Spec)
- https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md (configuration guide)
- The `plugins."io.containerd.grpc.v1.cri"` section does not apply to nerdctl, as nerdctl does not use CRI

35
src/virt/nerdctl/nerdctldocs/overlaybd.md Normal file
View File

@@ -0,0 +1,35 @@
# Lazy-pulling using OverlayBD Snapshotter
| :zap: Requirement | nerdctl >= 0.15.0 |
| ----------------- | --------------- |
OverlayBD is a remote container image format base on block-device which is an open-source implementation of paper ["DADI: Block-Level Image Service for Agile and Elastic Application Deployment. USENIX ATC'20".](https://www.usenix.org/conference/atc20/presentation/li-huiba)
See https://github.com/containerd/accelerated-container-image to learn further information.
## Enable lazy-pulling for `nerdctl run`
- Install containerd remote snapshotter plugin (`overlaybd`) from https://github.com/containerd/accelerated-container-image/blob/main/docs/BUILDING.md
- Add the following to `/etc/containerd/config.toml`:
```toml
[proxy_plugins]
[proxy_plugins.overlaybd]
type = "snapshot"
address = "/run/overlaybd-snapshotter/overlaybd.sock"
```
- Launch `containerd` and `overlaybd-snapshotter`
- Run `nerdctl` with `--snapshotter=overlaybd`
```console
nerdctl run --net host -it --rm --snapshotter=overlaybd registry.hub.docker.com/overlaybd/redis:6.2.1_obd
```
For more details about how to build overlaybd image, please refer to [accelerated-container-image](https://github.com/containerd/accelerated-container-image/blob/main/docs/IMAGE_CONVERTOR.md) conversion tool.
## Build OverlayBD image using `nerdctl image convert`
Nerdctl supports to convert an OCI image or docker format v2 image to OverlayBD image by using the `nerdctl image convert` command.
Before the conversion, you should have the `overlaybd-snapshotter` binary installed, which build from [accelerated-container-image](https://github.com/containerd/accelerated-container-image). You can run the command like `nerdctl image convert --overlaybd --oci <source_image> <target_image>` to convert the `<source_image>` to a OverlayBD image whose tag is `<target_image>`.

471
src/virt/nerdctl/nerdctldocs/registry.md Normal file
View File

@@ -0,0 +1,471 @@
# registry authentication
nerdctl uses `${DOCKER_CONFIG}/config.json` for the authentication with image registries.
`$DOCKER_CONFIG` defaults to `$HOME/.docker`.
## Using insecure registry
If you face `http: server gave HTTP response to HTTPS client` and you cannot configure TLS for the registry, try `--insecure-registry` flag:
e.g.,
```console
$ nerdctl --insecure-registry run --rm 192.168.12.34:5000/foo
```
## Specifying certificates
| :zap: Requirement | nerdctl >= 0.16 |
|-------------------|-----------------|
Create `~/.config/containerd/certs.d/<HOST:PORT>/hosts.toml` (or `/etc/containerd/certs.d/...` for rootful) to specify `ca` certificates.
```toml
# An example of ~/.config/containerd/certs.d/192.168.12.34:5000/hosts.toml
# (The path is "/etc/containerd/certs.d/192.168.12.34:5000/hosts.toml" for rootful)
server = "https://192.168.12.34:5000"
[host."https://192.168.12.34:5000"]
ca = "/path/to/ca.crt"
```
See https://github.com/containerd/containerd/blob/main/docs/hosts.md for the syntax of `hosts.toml` .
Docker-style directories are also supported.
The path is `~/.config/docker/certs.d` for rootless, `/etc/docker/certs.d` for rootful.
## Accessing 127.0.0.1 from rootless nerdctl
Currently, rootless nerdctl cannot pull images from 127.0.0.1, because
the pull operation occurs in RootlessKit's network namespace.
See https://github.com/containerd/nerdctl/issues/86 for the discussion about workarounds.
- - -
# Using managed registry services
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
- [Amazon Elastic Container Registry (ECR)](#amazon-elastic-container-registry-ecr)
- [Logging in](#logging-in)
- [Creating a repo](#creating-a-repo)
- [Pushing an image](#pushing-an-image)
- [Azure Container Registry (ACR)](#azure-container-registry-acr)
- [Creating a registry](#creating-a-registry)
- [Logging in](#logging-in-1)
- [Creating a repo](#creating-a-repo-1)
- [Pushing an image](#pushing-an-image-1)
- [Docker Hub](#docker-hub)
- [Logging in](#logging-in-2)
- [Creating a repo](#creating-a-repo-2)
- [Pushing an image](#pushing-an-image-2)
- [GitHub Container Registry (GHCR)](#github-container-registry-ghcr)
- [Logging in](#logging-in-3)
- [Creating a repo](#creating-a-repo-3)
- [Pushing an image](#pushing-an-image-3)
- [GitLab Container Registry](#gitlab-container-registry)
- [Logging in](#logging-in-4)
- [Creating a repo](#creating-a-repo-4)
- [Pushing an image](#pushing-an-image-4)
- [Google Artifact Registry (pkg.dev)](#google-artifact-registry-pkgdev)
- [Logging in](#logging-in-5)
- [Creating a repo](#creating-a-repo-5)
- [Pushing an image](#pushing-an-image-5)
- [Google Container Registry (GCR) [DEPRECATED]](#google-container-registry-gcr-deprecated)
- [Logging in](#logging-in-6)
- [Creating a repo](#creating-a-repo-6)
- [Pushing an image](#pushing-an-image-6)
- [JFrog Artifactory (Cloud/On-Prem)](#jfrog-artifactory-cloudon-prem)
- [Logging in](#logging-in-7)
- [Creating a repo](#creating-a-repo-7)
- [Pushing an image](#pushing-an-image-7)
- [Quay.io](#quayio)
- [Logging in](#logging-in-8)
- [Creating a repo](#creating-a-repo-8)
- [Pushing an image](#pushing-an-image-8)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Amazon Elastic Container Registry (ECR)
See also https://aws.amazon.com/ecr
### Logging in
```console
$ aws ecr get-login-password --region <REGION> | nerdctl login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com
Login Succeeded
```
<details>
<summary>Alternative method: <code>docker-credential-ecr-login</code></summary>
This methods is more secure but needs an external dependency.
<p>
Install `docker-credential-ecr-login` from https://github.com/awslabs/amazon-ecr-credential-helper , and create the following files:
`~/.docker/config.json`:
```json
{
"credHelpers": {
"public.ecr.aws": "ecr-login",
"<AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com": "ecr-login"
}
}
```
`~/.aws/credentials`:
```
[default]
aws_access_key_id = ...
aws_secret_access_key = ...
```
> **Note**: If you are running nerdctl inside a VM (including Lima, Colima, Rancher Desktop, and WSL2), `docker-credential-ecr-login` has to be installed inside the guest, not the host.
> Same applies to the path of `~/.docker/config.json` and `~/.aws/credentials`, too.
</p>
</details>
### Creating a repo
You have to create a repository via https://console.aws.amazon.com/ecr/home/ .
### Pushing an image
```console
$ nerdctl tag hello-world <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/<REPO>
$ nerdctl push <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/<REPO>
```
The pushed image appears in the repository you manually created in the previous step.
## Azure Container Registry (ACR)
See also https://azure.microsoft.com/en-us/services/container-registry/#overview
### Creating a registry
You have to create a "Container registry" resource manually via [the Azure portal](https://portal.azure.com/).
### Logging in
```console
$ nerdctl login -u <USERNAME> <REGISTRY>.azurecr.io
Enter Password: ********[Enter]
Login Succeeded
```
The login credentials can be found as "Access keys" in [the Azure portal](https://portal.azure.com/).
See also https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication .
> **Note**: nerdctl prior to v0.16.1 had a bug that required pressing the Enter key twice.
### Creating a repo
You do not need to create a repo explicitly.
### Pushing an image
```console
$ nerdctl tag hello-world <REGISTRY>.azurecr.io/hello-world
$ nerdctl push <REGISTRY>.azurecr.io/hello-world
```
The pushed image appears in [the Azure portal](https://portal.azure.com/).
Private as default.
## Docker Hub
See also https://hub.docker.com/
### Logging in
```console
$ nerdctl login -u <USERNAME>
Enter Password: ********[Enter]
Login Succeeded
```
> **Note**: nerdctl prior to v0.16.1 had a bug that required pressing the Enter key twice.
### Creating a repo
You do not need to create a repo explicitly, for public images.
To create a private repo, see https://hub.docker.com/repositories .
### Pushing an image
```console
$ nerdctl tag hello-world <USERNAME>/hello-world
$ nerdctl push <USERNAME>/hello-world
```
The pushed image appears in https://hub.docker.com/repositories .
**Public** by default.
## GitHub Container Registry (GHCR)
See also https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
### Logging in
```console
$ nerdctl login ghcr.io -u <USERNAME>
Enter Password: ********[Enter]
Login Succeeded
```
The `<USERNAME>` is your GitHub username but in lower characters.
The "Password" here is a [GitHub Personal access token](https://github.com/settings/tokens), with `read:packages` and `write:packages` scopes.
> **Note**: nerdctl prior to v0.16.1 had a bug that required pressing the Enter key twice.
### Creating a repo
You do not need to create a repo explicitly.
### Pushing an image
```console
$ nerdctl tag hello-world ghcr.io/<USERNAME>/hello-world
$ nerdctl push ghcr.io/<USERNAME>/hello-world
```
The pushed image appears in the "Packages" tab of your GitHub profile.
Private as default.
## GitLab Container Registry
See also https://docs.gitlab.com/ee/user/packages/container_registry/
### Logging in
```console
$ nerdctl login registry.gitlab.com -u <USERNAME>
Enter Password: ********[Enter]
Login Succeeded
```
The `<USERNAME>` is your GitLab username.
The "Password" here is either a [GitLab Personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) or a [GitLab Deploy token](https://docs.gitlab.com/ee/user/project/deploy_tokens/index.html). Both options require minimum scope of `read_registry` for pull access and both `write_registry` and `read_registry` scopes for push access.
> **Note**: nerdctl prior to v0.16.1 had a bug that required pressing the Enter key twice.
### Creating a repo
Container registries in GitLab are created at the project level. A project in GitLab must exist first before you begin working with its container registry.
### Pushing an image
In this example we have created a GitLab project named `myproject`.
```console
$ nerdctl tag hello-world registry.gitlab.com/<USERNAME>/myproject/hello-world:latest
$ nerdctl push registry.gitlab.com/<USERNAME>/myproject/hello-world:latest
```
The pushed image appears under the "Packages & Registries -> Container Registry" tab of your project on GitLab.
## Google Artifact Registry (pkg.dev)
See also https://cloud.google.com/artifact-registry/docs/docker/quickstart
### Logging in
Create a [GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating), grant
`Artifact Registry Reader` and `Artifact Registry Writer` roles, and download the key as a JSON file.
Then run the following command:
```console
$ cat <GCP_SERVICE_ACCOUNT_KEY_JSON> | nerdctl login -u _json_key --password-stdin https://<REGION>-docker.pkg.dev
WARNING! Your password will be stored unencrypted in /home/<USERNAME>/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
```
See also https://cloud.google.com/artifact-registry/docs/docker/authentication
<details>
<summary>Alternative method: <code>docker-credential-gcloud</code> (<code>gcloud auth configure-docker</code>)</summary>
This methods is more secure but needs an external dependency.
<p>
Run `gcloud auth configure-docker <REGION>-docker.pkg.dev`, e.g.,
```console
$ gcloud auth configure-docker asia-northeast1-docker.pkg.dev
Adding credentials for: asia-northeast1-docker.pkg.dev
After update, the following will be written to your Docker config file located at [/home/<USERNAME>/.docker/config.json]:
{
"credHelpers": {
"asia-northeast1-docker.pkg.dev": "gcloud"
}
}
Do you want to continue (Y/n)? y
Docker configuration file updated.
```
Google Cloud SDK (`gcloud`, `docker-credential-gcloud`) has to be installed, see https://cloud.google.com/sdk/docs/quickstart .
> **Note**: If you are running nerdctl inside a VM (including Lima, Colima, Rancher Desktop, and WSL2), the Google Cloud SDK has to be installed inside the guest, not the host.
</p>
</details>
### Creating a repo
You have to create a repository via https://console.cloud.google.com/artifacts .
Choose "Docker" as the repository format.
### Pushing an image
```console
$ nerdctl tag hello-world <REGION>-docker.pkg.dev/<GCP_PROJECT_ID>/<REPO>/hello-world
$ nerdctl push <REGION>-docker.pkg.dev/<GCP_PROJECT_ID>/<REPO>/hello-world
```
The pushed image appears in the repository you manually created in the previous step.
## Google Container Registry (GCR) [DEPRECATED]
See also https://cloud.google.com/container-registry/docs/advanced-authentication
### Logging in
Create a [GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating), grant
`Storage Object Admin` role, and download the key as a JSON file.
Then run the following command:
```console
$ cat <GCP_SERVICE_ACCOUNT_KEY_JSON> | nerdctl login -u _json_key --password-stdin https://asia.gcr.io
WARNING! Your password will be stored unencrypted in /home/<USERNAME>/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
```
See also https://cloud.google.com/container-registry/docs/advanced-authentication
<details>
<summary>Alternative method: <code>docker-credential-gcloud</code> (<code>gcloud auth configure-docker</code>)</summary>
This methods is more secure but needs an external dependency.
<p>
```console
$ gcloud auth configure-docker
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
After update, the following will be written to your Docker config file located at [/home/<USERNAME>/.docker/config.json]:
{
"credHelpers": {
"gcr.io": "gcloud",
"us.gcr.io": "gcloud",
"eu.gcr.io": "gcloud",
"asia.gcr.io": "gcloud",
"staging-k8s.gcr.io": "gcloud",
"marketplace.gcr.io": "gcloud"
}
}
Do you want to continue (Y/n)? y
Docker configuration file updated.
```
Google Cloud SDK (`gcloud`, `docker-credential-gcloud`) has to be installed, see https://cloud.google.com/sdk/docs/quickstart .
> **Note**: If you are running nerdctl inside a VM (including Lima, Colima, Rancher Desktop, and WSL2), the Google Cloud SDK has to be installed inside the guest, not the host.
</p>
</details>
### Creating a repo
You do not need to create a repo explicitly.
### Pushing an image
```console
$ nerdctl tag hello-world asia.gcr.io/<GCP_PROJECT_ID>/hello-world
$ nerdctl push asia.gcr.io/<GCP_PROJECT_ID>/hello-world
```
The pushed image appears in https://console.cloud.google.com/gcr/ .
Private by default.
## JFrog Artifactory (Cloud/On-Prem)
See also https://www.jfrog.com/confluence/display/JFROG/Getting+Started+with+Artifactory+as+a+Docker+Registry
### Logging in
```console
$ nerdctl login <SERVER_NAME>.jfrog.io -u <USERNAME>
Enter Password: ********[Enter]
Login Succeeded
```
Login using the default username: admin, and password: password for the on-prem installation, or the credentials provided to you by email for the cloud installation.
JFrog Platform is integrated with OAuth allowing you to delegate authentication requests to external providers (the provider types supported are Google, OpenID Connect, GitHub Enterprise, and Cloud Foundry UAA)
> **Note**: nerdctl prior to v0.16.1 had a bug that required pressing the Enter key twice.
### Creating a repo
1. Add local Docker repository
1. Add a new Local Repository with the Docker package type via `https://<server-name>.jfrog.io/ui/admin/repositories/local/new`.
2. Add virtual Docker repository
1. Add a new virtual repository with the Docker package type via `https://<server-name>.jfrog.io/ui/admin/repositories/virtual/new`.
2. Add the local docker repository you created in Steps 1 (move it from Available Repositories to Selected Repositories using the arrow buttons).
3. Set local repository as a default local deployment repository.
### Pushing an image
```console
$ nerdctl tag hello-world <SERVER_NAME>.jfrog.io/<VIRTUAL_REPO_NAME>/hello-world
$ nerdctl push <SERVER_NAME>.jfrog.io/<VIRTUAL_REPO_NAME>/hello-world
```
The `SERVER_NAME` is the first part of the URL given to you for your environment: `https://<SERVER_NAME>.jfrog.io`
The `VIRTUAL_REPO_NAME` is the name “docker” that you assigned to your virtual repository in 2.i .
The pushed image appears in `https://<SERVER_NAME>.jfrog.io/ui/repos/tree/General/<VIRTUAL_REPO_NAME>` .
Private by default.
## Quay.io
See also https://docs.quay.io/solution/getting-started.html
### Logging in
```console
$ nerdctl login quay.io -u <USERNAME>
Enter Password: ********[Enter]
Login Succeeded
```
> **Note**: nerdctl prior to v0.16.1 had a bug that required pressing the Enter key twice.
### Creating a repo
You do not need to create a repo explicitly.
### Pushing an image
```console
$ nerdctl tag hello-world quay.io/<USERNAME>/hello-world
$ nerdctl push quay.io/<USERNAME>/hello-world
```
The pushed image appears in https://quay.io/repository/ .
Private as default.

193
src/virt/nerdctl/nerdctldocs/rootless.md Normal file
View File

@@ -0,0 +1,193 @@
# Rootless mode
See https://rootlesscontaine.rs/getting-started/common/ for the prerequisites.
## Daemon (containerd)
Use [`containerd-rootless-setuptool.sh`](../extras/rootless) to set up rootless containerd.
```console
$ containerd-rootless-setuptool.sh install
[INFO] Checking RootlessKit functionality
[INFO] Checking cgroup v2
[INFO] Checking overlayfs
[INFO] Creating /home/testuser/.config/systemd/user/containerd.service
...
[INFO] Installed containerd.service successfully.
[INFO] To control containerd.service, run: `systemctl --user (start|stop|restart) containerd.service`
[INFO] To run containerd.service on system startup, run: `sudo loginctl enable-linger testuser`
[INFO] Use `nerdctl` to connect to the rootless containerd.
[INFO] You do NOT need to specify $CONTAINERD_ADDRESS explicitly.
```
The usage of `containerd-rootless-setuptool.sh` is almost same as [`dockerd-rootless-setuptool.sh`](https://rootlesscontaine.rs/getting-started/docker/) .
Resource limitation flags such as `nerdctl run --memory` require systemd and cgroup v2: https://rootlesscontaine.rs/getting-started/common/cgroup2/
#### AppArmor Profile for Ubuntu 24.04+
Configuring AppArmor is needed only on Ubuntu 24.04+, with RootlessKit installed under a non-standard path: https://rootlesscontaine.rs/getting-started/common/apparmor/
## Client (nerdctl)
Just execute `nerdctl`. No need to specify the socket address manually.
```console
$ nerdctl run -it --rm alpine
```
Depending on your kernel version, you may need to enable FUSE-OverlayFS or set `export CONTAINERD_SNAPSHOTTER=native`.
(See below.)
## Add-ons
### BuildKit
To enable BuildKit, run the following command:
```console
$ containerd-rootless-setuptool.sh install-buildkit
```
## Snapshotters
### OverlayFS
The default `overlayfs` snapshotter only works on the following hosts:
- Any distro, with kernel >= 5.13
- Non-SELinux distro, with kernel >= 5.11
- Ubuntu since 2015
For other hosts, [`fuse-overlayfs` snapshotter](https://github.com/containerd/fuse-overlayfs-snapshotter) needs to be used instead.
### FUSE-OverlayFS
To enable `fuse-overlayfs` snapshotter, run the following command:
```console
$ containerd-rootless-setuptool.sh install-fuse-overlayfs
```
Then, add the following config to `~/.config/containerd/config.toml`, and run `systemctl --user restart containerd.service`:
```toml
[proxy_plugins]
[proxy_plugins."fuse-overlayfs"]
type = "snapshot"
# NOTE: replace "1000" with your actual UID
address = "/run/user/1000/containerd-fuse-overlayfs.sock"
```
The snapshotter can be specified as `$CONTAINERD_SNAPSHOTTER`.
```console
$ export CONTAINERD_SNAPSHOTTER=fuse-overlayfs
$ nerdctl run -it --rm alpine
```
If `fuse-overlayfs` does not work, try `export CONTAINERD_SNAPSHOTTER=native`.
### Stargz Snapshotter
[Stargz Snapshotter](./stargz.md) enables lazy-pulling of images.
To enable Stargz snapshotter, run the following command:
```console
$ containerd-rootless-setuptool.sh install-stargz
```
Then, add the following config to `~/.config/containerd/config.toml` and run `systemctl --user restart containerd.service`:
```toml
[proxy_plugins]
[proxy_plugins."stargz"]
type = "snapshot"
# NOTE: replace "1000" with your actual UID
address = "/run/user/1000/containerd-stargz-grpc/containerd-stargz-grpc.sock"
```
The snapshotter can be specified as `$CONTAINERD_SNAPSHOTTER`.
```console
$ export CONTAINERD_SNAPSHOTTER=stargz
$ nerdctl run -it --rm ghcr.io/stargz-containers/alpine:3.10.2-esgz
```
See https://github.com/containerd/stargz-snapshotter/blob/main/docs/pre-converted-images.md for the image list.
## bypass4netns
| :zap: Requirement | nerdctl >= 0.17 |
|-------------------|-----------------|
[bypass4netns](https://github.com/rootless-containers/bypass4netns) is an accelerator for rootless networking.
This improves **outgoing or incoming (with --publish option) networking performance.**
The performance benchmark with iperf3 on Ubuntu 21.10 on Hyper-V VM is shown below.
| iperf3 benchmark | without bypass4netns | with bypass4netns |
| ----------------- | -------------------- | ----------------- |
| container -> host | 0.398 Gbps | **42.2 Gbps** |
| host -> container | 20.6 Gbps | **47.4 Gbps** |
This benchmark can be reproduced with [https://github.com/rootless-containers/bypass4netns/blob/f009d96139e9e38ce69a2ea8a9a746349bad273c/Vagrantfile](https://github.com/rootless-containers/bypass4netns/blob/f009d96139e9e38ce69a2ea8a9a746349bad273c/Vagrantfile)
Acceleration with bypass4netns is available with:
- `--annotation nerdctl/bypass4netns=true` (for nerdctl v2.0 and later)
- `--label nerdctl/bypass4netns=true` (deprecated form, used in nerdctl prior to v2.0).
You also need to have `bypass4netnsd` (bypass4netns daemon) to be running.
Example
```console
$ containerd-rootless-setuptool.sh install-bypass4netnsd
$ nerdctl run -it --rm -p 8080:80 --annotation nerdctl/bypass4netns=true alpine
```
More detail is available at [https://github.com/rootless-containers/bypass4netns/blob/master/README.md](https://github.com/rootless-containers/bypass4netns/blob/master/README.md)
## Configuring RootlessKit
Rootless containerd recognizes the following environment variables to configure the behavior of [RootlessKit](https://github.com/rootless-containers/rootlesskit):
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_STATE_DIR=DIR`: the rootlesskit state dir. Defaults to `$XDG_RUNTIME_DIR/containerd-rootless`.
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_NET=(slirp4netns|vpnkit|lxc-user-nic)`: the rootlesskit network driver. Defaults to "slirp4netns" if slirp4netns (>= v0.4.0) is installed. Otherwise defaults to "vpnkit".
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_MTU=NUM`: the MTU value for the rootlesskit network driver. Defaults to 65520 for slirp4netns, 1500 for other drivers.
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=(builtin|slirp4netns)`: the rootlesskit port driver. Defaults to "builtin" (this driver does not propagate the container's source IP address and always uses 127.0.0.1. Please check [Port Drivers](https://github.com/rootless-containers/rootlesskit/blob/master/docs/port.md#port-drivers) for more details).
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_SLIRP4NETNS_SANDBOX=(auto|true|false)`: whether to protect slirp4netns with a dedicated mount namespace. Defaults to "auto".
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_SLIRP4NETNS_SECCOMP=(auto|true|false)`: whether to protect slirp4netns with seccomp. Defaults to "auto".
* `CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=(auto|true|false)`: whether to launch rootlesskit with the "detach-netns" mode.
Defaults to "auto", which is resolved to "true" if RootlessKit >= 2.0 is installed.
The "detached-netns" mode accelerates `nerdctl (pull|push|build)` and enables `nerdctl run --net=host`,
however, there is a relatively minor drawback with BuildKit prior to v0.13:
the host loopback IP address (127.0.0.1) and abstract sockets are exposed to Dockerfile's "RUN" instructions during `nerdctl build` (not `nerdctl run`).
The drawback is fixed in BuildKit v0.13. Upgrading from a prior version of BuildKit needs removing the old systemd unit:
`containerd-rootless-setuptool.sh uninstall-buildkit && rm -f ~/.config/buildkit/buildkitd.toml`
To set these variables, create `~/.config/systemd/user/containerd.service.d/override.conf` as follows:
```ini
[Service]
Environment=CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS="false"
```
And then run the following commands:
```bash
systemctl --user daemon-reload
systemctl --user restart containerd
```
## Troubleshooting
### Hint to Fedora users
- If SELinux is enabled on your host and your kernel is older than 5.13, you need to use [`fuse-overlayfs` instead of `overlayfs`](#fuse-overlayfs).
## Rootlesskit Network Design
In `detach-netns` mode:
- Network namespace is detached and stored in `$ROOTLESSKIT_STATE_DIR/netns`.
- The child command executes within the host's network namespace, allowing actions like `pull` and `push` to happen in the host network namespace.
- For creating and configuring the container's network namespace, the child command switches temporarily to the relevant namespace located in `$ROOTLESSKIT_STATE_DIR/netns`. This ensures necessary network setup while maintaining isolation in the host namespace.
![rootlessKit-network-design.png](images/rootlessKit-network-design.png)
- Rootlesskit Parent NetNS and Child NetNS are already configured by the startup script [containerd-rootless.sh](https://github.com/containerd/nerdctl/blob/main/extras/rootless/containerd-rootless.sh)
- Rootlesskit Parent NetNS is the host network namespace
- step1: `nerdctl` calls `containerd` in the host network namespace.
- step2: `containerd` calls `runc` in the host network namespace.
- step3: `runc` creates container with dedicated namespaces (e.g network ns) in the Parent netns.
- step4: `runc` nsenter Rootlesskit Child NetNS before triggering nerdctl ocihook.
- step5: `nerdctl` ocihook module leverages CNI.
- step6: CNI configures container network namespace: create network interfaces `eth0` -> `veth0` -> `nerdctl0`.

47
src/virt/nerdctl/nerdctldocs/soci.md Normal file
View File

@@ -0,0 +1,47 @@
# Lazy-pulling using SOCI Snapshotter
SOCI Snapshotter is a containerd snapshotter plugin. It enables standard OCI images to be lazily loaded without requiring a build-time conversion step. "SOCI" is short for "Seekable OCI", and is pronounced "so-CHEE".
See https://github.com/awslabs/soci-snapshotter to learn further information.
## Prerequisites
- Install containerd remote snapshotter plugin (`soci-snapshotter-grpc`) from https://github.com/awslabs/soci-snapshotter/blob/main/docs/getting-started.md
- Add the following to `/etc/containerd/config.toml`:
```toml
[proxy_plugins]
[proxy_plugins.soci]
type = "snapshot"
address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"
```
- Launch `containerd` and `soci-snapshotter-grpc`
## Enable SOCI for `nerdctl run` and `nerdctl pull`
| :zap: Requirement | nerdctl >= 1.5.0 |
| ----------------- | ---------------- |
- Run `nerdctl` with `--snapshotter=soci`
```console
nerdctl run -it --rm --snapshotter=soci public.ecr.aws/soci-workshop-examples/ffmpeg:latest
```
- You can also only pull the image with SOCI without running the container.
```console
nerdctl pull --snapshotter=soci public.ecr.aws/soci-workshop-examples/ffmpeg:latest
```
For images that already have SOCI indices, see https://gallery.ecr.aws/soci-workshop-examples
## Enable SOCI for `nerdctl push`
| :zap: Requirement | nerdctl >= 1.6.0 |
| ----------------- | ---------------- |
- Push the image with SOCI index. Adding `--snapshotter=soci` arg to `nerdctl pull`, `nerdctl` will create the SOCI index and push the index to same destination as the image.
```console
nerdctl push --snapshotter=soci --soci-span-size=2097152 --soci-min-layer-size=20971520 public.ecr.aws/my-registry/my-repo:latest
```
--soci-span-size and --soci-min-layer-size are two properties to customize the SOCI index. See [Command Reference](https://github.com/containerd/nerdctl/blob/377b2077bb616194a8ef1e19ccde32aa1ffd6c84/docs/command-reference.md?plain=1#L773) for further details.

187
src/virt/nerdctl/nerdctldocs/stargz.md Normal file
View File

@@ -0,0 +1,187 @@
# Lazy-pulling using Stargz Snapshotter
| :zap: Requirement | nerdctl >= 0.0.1 |
|-------------------|------------------|
Lazy-pulling is a technique to running containers before completion of pulling the images.
See https://github.com/containerd/stargz-snapshotter to learn further information.
[![asciicast](https://asciinema.org/a/378377.svg)](https://asciinema.org/a/378377)
## Enable lazy-pulling for `nerdctl run`
> **NOTE**
> For rootless installation, see [`rootless.md`](./rootless.md#stargz-snapshotter)
- Install Stargz plugin (`containerd-stargz-grpc`) from https://github.com/containerd/stargz-snapshotter
- Add the following to `/etc/containerd/config.toml`:
```toml
[proxy_plugins]
[proxy_plugins.stargz]
type = "snapshot"
address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"
```
- Launch `containerd` and `containerd-stargz-grpc`
- Run `nerdctl` with `--snapshotter=stargz`
```console
# nerdctl --snapshotter=stargz run -it --rm ghcr.io/stargz-containers/fedora:30-esgz
```
For the list of pre-converted Stargz images, see https://github.com/containerd/stargz-snapshotter/blob/main/docs/pre-converted-images.md
### Benchmark result (Dec 9, 2020)
For running `python3 -c print("hi")`, eStargz with Stargz Snapshotter is 3-4 times faster than the legacy OCI with overlayfs snapshotter.
Legacy OCI with overlayfs snapshotter:
```console
# time nerdctl --snapshotter=overlayfs run -it --rm ghcr.io/stargz-containers/python:3.7-org python3 -c 'print("hi")'
ghcr.io/stargz-containers/python:3.7-org: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:6008006c63b0a6043a11ac151cee572e0c8676b4ba3130ff23deff5f5d711237: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:48eafda05f80010a6677294473d51a530e8f15375b6447195b6fb04dc2a30ce7: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f860607a6cd9751ac8db2f33cbc3ce1777a44eb3c04853e116763441a304fbf6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:96b2c1e36db5f5910f58da2ca4f9311b0690810c7107fb055ee1541498b5061f: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c495e8de12d26c9843a7a2bf8c68de1e5652e66d80d9bc869279f9af6f86736a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:33382189822a108b249cf3ccd234d04c3a8dfe7d593df19c751dcfab3675d5f2: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:94c9a318e47ab8a318582e2712bb495f92f17a7c1e50f13cc8a3e362c1b09290: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6eaa0b6b8562fb4a02e140ae53b3910fc4d0db6e68660390eaef993f42e21102: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:adbdcbacafe93bf0791e49c8d3689bb78d9e60d02d384d4e14433aedae39f52c: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:756975cb9c7e7933d824af9319b512dd72a50894232761d06ef3be59981df838: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d77915b4e630d47296770ce4cf481894885978072432456615172af463433cc5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5f37a0a41b6b03489dd7de0aa2a79e369fd8b219bbc36b52f3f9790dc128e74b: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 41.9s total: 321.3 (7.7 MiB/s)
hi
real 0m51.754s
user 0m2.687s
sys 0m5.533s
```
eStargz with Stargz Snapshotter:
```console
# time nerdctl --snapshotter=stargz run -it --rm ghcr.io/stargz-containers/python:3.7-esgz python3 -c 'print("hi")'
fetching sha256:2ea0dd96... application/vnd.oci.image.index.v1+json
fetching sha256:9612ff73... application/vnd.docker.distribution.manifest.v2+json
fetching sha256:34e5920e... application/vnd.docker.container.image.v1+json
hi
real 0m13.589s
user 0m0.132s
sys 0m0.158s
```
## Enable lazy-pulling for pulling base images during `nerdctl build`
- Launch `buildkitd` with `--oci-worker-snapshotter=stargz` (or `--containerd-worker-snapshotter=stargz` if you use containerd worker)
- Launch `nerdctl build`. No need to specify `--snapshotter` for `nerdctl`.
## Building stargz images using `nerdctl build`
```console
$ nerdctl build -t example.com/foo .
$ nerdctl image convert --estargz --oci example.com/foo example.com/foo:estargz
$ nerdctl push example.com/foo:estargz
```
NOTE: `--estargz` should be specified in conjunction with `--oci`
Stargz Snapshotter is not needed for building stargz images.
## Tips for image conversion
### Tips 1: Creating smaller eStargz images
`nerdctl image convert` allows the following flags for optionally creating a smaller eStargz image.
The result image requires stargz-snapshotter >= v0.13.0 for lazy pulling.
- `--estargz-min-chunk-size`: The minimal number of bytes of data must be written in one gzip stream. If it's > 0, multiple files and chunks can be written into one gzip stream. Smaller number of gzip header and smaller size of the result blob can be expected. `--estargz-min-chunk-size=0` produces normal eStargz.
- `--estargz-external-toc`: Separate TOC JSON metadata into another image (called "TOC image"). The result eStargz doesn't contain TOC so we can expect a smaller size than normal eStargz. This is an [experimental](./experimental.md) feature.
#### `--estargz-min-chunk-size` usage
conversion:
```console
# nerdctl image convert --oci --estargz --estargz-min-chunk-size=50000 ghcr.io/stargz-containers/ubuntu:22.04 registry2:5000/ubuntu:22.04-chunk50000
# nerdctl image ls
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
ghcr.io/stargz-containers/ubuntu 22.04 20fa2d7bb4de 14 seconds ago linux/amd64 83.4 MiB 29.0 MiB
registry2:5000/ubuntu 22.04-chunk50000 562e09e1b3c1 2 seconds ago linux/amd64 0.0 B 29.2 MiB
# nerdctl push --insecure-registry registry2:5000/ubuntu:22.04-chunk50000
```
Pull it lazily:
```console
# nerdctl pull --snapshotter=stargz --insecure-registry registry2:5000/ubuntu:22.04-chunk50000
# mount | grep "stargz on"
stargz on /var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)
```
#### `--estargz-external-toc` usage
convert:
```console
# nerdctl image convert --oci --estargz --estargz-external-toc ghcr.io/stargz-containers/ubuntu:22.04 registry2:5000/ubuntu:22.04-ex
INFO[0005] Extra image(0) registry2:5000/ubuntu:22.04-ex-esgztoc
sha256:3059dd5d9c404344e0b7c43d9782de8cae908531897262b7772103a0b585bbee
# nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
ghcr.io/stargz-containers/ubuntu 22.04 20fa2d7bb4de 9 seconds ago linux/amd64 83.4 MiB 29.0 MiB
registry2:5000/ubuntu 22.04-ex 3059dd5d9c40 1 second ago linux/amd64 0.0 B 30.8 MiB
registry2:5000/ubuntu 22.04-ex-esgztoc 18c042b6eb8b 1 second ago linux 0.0 B 151.3 KiB
```
Then push eStargz(`registry2:5000/ubuntu:22.04-ex`) and TOC image(`registry2:5000/ubuntu:22.04-ex-esgztoc`) to the same registry (`registry2` is used in this example but you can use arbitrary registries):
```console
# nerdctl push --insecure-registry registry2:5000/ubuntu:22.04-ex
# nerdctl push --insecure-registry registry2:5000/ubuntu:22.04-ex-esgztoc
```
Pull it lazily:
```console
# nerdctl pull --insecure-registry --snapshotter=stargz registry2:5000/ubuntu:22.04-ex
```
Stargz Snapshotter automatically refers to the TOC image on the same registry.
##### optional `--estargz-keep-diff-id` flag for conversion without changing layer diffID
`nerdctl image convert` supports optional flag `--estargz-keep-diff-id` specified with `--estargz-external-toc`.
This converts an image to eStargz without changing the diffID (uncompressed digest) so even eStargz-agnostic gzip decompressor (e.g. gunzip) can restore the original tar blob.
```console
# nerdctl image convert --oci --estargz --estargz-external-toc --estargz-keep-diff-id ghcr.io/stargz-containers/ubuntu:22.04 registry2:5000/ubuntu:22.04-ex-keepdiff
# nerdctl push --insecure-registry registry2:5000/ubuntu:22.04-ex-keepdiff
# nerdctl push --insecure-registry registry2:5000/ubuntu:22.04-ex-keepdiff-esgztoc
# crane --insecure blob registry2:5000/ubuntu:22.04-ex-keepdiff@sha256:2dc39ba059dcd42ade30aae30147b5692777ba9ff0779a62ad93a74de02e3e1f | jq -r '.rootfs.diff_ids[]'
sha256:7f5cbd8cc787c8d628630756bcc7240e6c96b876c2882e6fc980a8b60cdfa274
# crane blob ghcr.io/stargz-containers/ubuntu:22.04@sha256:2dc39ba059dcd42ade30aae30147b5692777ba9ff0779a62ad93a74de02e3e1f | jq -r '.rootfs.diff_ids[]'
sha256:7f5cbd8cc787c8d628630756bcc7240e6c96b876c2882e6fc980a8b60cdfa274
```
### Tips 2: Using zstd instead of gzip (a.k.a. zstd:chunked)
You can use zstd compression with lazy pulling support (a.k.a zstd:chunked) instead of gzip.
- Pros
- [Faster](https://github.com/facebook/zstd/tree/v1.5.2#benchmarks) compression/decompression.
- Cons
- Old tools might not support. And unsupported by some tools yet.
- zstd supported by OCI Image Specification is still under rc (2022/11). will be added to [v1.1.0](https://github.com/opencontainers/image-spec/commit/1a29e8675a64a5cdd2d93b6fa879a82d9a4d926a).
- zstd supported by [docker >=v23.0.0](https://github.com/moby/moby/releases/tag/v23.0.0).
- zstd supported by [containerd >= v1.5](https://github.com/containerd/containerd/releases/tag/v1.5.0).
- `min-chunk-size`, `external-toc` (described in Tips 1) are unsupported yet.
```console
$ nerdctl build -t example.com/foo .
$ nerdctl image convert --zstdchunked --oci example.com/foo example.com/foo:zstdchunked
$ nerdctl push example.com/foo:zstdchunked
```