unpackOptions := &layer.UnpackOptions{
    MapOptions: mapOptions,
    WhiteoutMode: layer.StandardOCIWhiteout, // or layer.OverlayFSWhiteout
}
err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)

will have to now be written as

unpackOptions := &layer.UnpackOptions{
    OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
        MapOptions: mapOptions,
    },
}
err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)

and similarly

repackOptions := &layer.RepackOptions{
    MapOptions: mapOptions,
    TranslateOverlayWhiteouts: false, // or true
}
layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)

will have to now be written as

repackOptions := &layer.RepackOptions{
    OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
        MapOptions: mapOptions,
    },
}
layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)

Note that this means you can easily re-use the OnDiskFormat configuration between both UnpackOptions and RepackOptions, removing the previous need to translate between WhiteoutMode and TranslateOverlayWhiteouts.

For users of the API that need to extract the MapOptions from UnpackOptions and RepackOptions, there is a new helper MapOptions which will help extract it without doing interface type switching. For OnDiskFormat there is also a Map method that gives you the inner MapOptions regardless of type.

Added

• umoci unpack now supports handling layers compressed with zstd. This is something that was added in image-spec v1.2 (which we do not yet support fully) but at least this will allow users to operate on zstd-compressed images, which are slowly becoming more common.
• umoci repack and umoci insert now support creating zstd-compressed layers. The default behaviour (called auto) is to try to match the last layer's compression algorithm, with a fallback to gzip if none of the layer algorithms were supported.
  • Users can specify their preferred compression algorithm using the new --compress flag. You can also disable compression entirely using --compress=none but --compress=auto will never automatically choose none compression.
• GenerateLayer and GenerateInsertLayer with OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci 0.4.7) now support converting trusted.overlay.opaque=y and trusted.overlay.whiteout whiteouts into OCI whiteouts when generating OCI layers.
• OverlayfsRootfs now supports compatibility with the userxattr mount option for overlayfs (where user.overlay.* xattrs are used rather than the default trusted.overlay.*). This is a pretty key compatibility feature for users that use unprivileged overlayfs mounts and will hopefully remove the need for most downstream forks hacking in this functionality (such as stacker). For Go API users, to enable this just set UserXattr: true in OverlayfsRootfs. Note that (as with upstream overlayfs), only one xattr namespace is ever used (so if OverlayfsRootfs.UserXattr == true then trusted.overlay.* xattrs will be treated like any other non-overlayfs xattr).

Changes

• In this release, the primary development branch was renamed to main.
• The runtime-spec version of the config.json version we generate is no longer hard-coded to 1.0.0. We now use the version of the spec we have imported (with any -dev suffix stripped, as such a prefix causes havoc with verification tools -- ideally we would only ever use released versions of the spec but that's not always possible). #452
• Add the cgroup namespace to the default configuration generated by umoci unpack to make sure that our configuration plays nicely with runc when on cgroupv2 systems.
• umoci has been migrated away from github.com/pkg/errors to Go stdlib error wrapping.
• The gzip compression block size has been updated to be more friendly with Docker and other tools that might round-trip the layer blob data (causing the hash to change if the block size is different). #509

Fixed

• In 0.4.7, a performance regression was introduced as part of the VerifiedReadCloser hardening work (to read all trailing bytes) which would cause walk operations on images to hash every blob in the image (even blobs which we couldn't parse and thus couldn't recurse into). To resolve this, we no longer recurse into unparseable blobs. #373 #375 #394
• Handle EINTR on io.Copy operations. Newer Go versions have added more opportunistic pre-emption which can cause EINTR errors in io paths that didn't occur before. #437
• Quite a few changes were made to CI to try to avoid issues with fragility. #452
• umoci will now return an explicit error if you pass invalid uid or gid values to --uid-map and --gid-map rather than silently truncating the value.
• For Go users of umoci, GenerateLayer (but not GenerateInsertLayer) with OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci 0.4.7) had several severe bugs that made the feature unusable:
  • All OCI whiteouts added to the archive would incorrectly have the full host name of the path rather than the correctly rooted path, making the whiteout practically useless.
  • Any non-whiteout files would not be included in the layer, making the layer data incomplete and thus resulting in silent data loss. Given how severe these bugs were and the lack of bug reports of this issue in the past 4 years, it seems this feature has not really been used by anyone (I hope...).
• For Go users of umoci, UnpackLayer now correctly handles several aspects of OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7) extraction that weren't handled correctly:
  • Unlike regular extractions, overlayfs-style extractions require us to create the parent directory of the whiteout (rather than ignoring or assuming the underlying path exists) because the whiteout is being created in a separate layer to the underlying file. We also need to make sure that opaque whiteout targets are directories.
  • trusted.overlay.opaque=y has very peculiar behaviour when a regular whiteout (i.e. mknod c 0 0) is placed inside an opaque directory -- the whiteout-ed file appears in readdir but the file itself doesn't exist. To avoid this confusion (and possible information leak), umoci will no longer extract plain whiteouts within an opaque whiteout directory in the same layer. (As per the OCI spec requirements, this is regardless of the order of the opaque whiteout and the regular whiteout in the layer archive.)
• UnpackLayer and Generate(Insert)Layer now correctly handle trusted.overlay.* xattr escaping when extracting and generating layers with the overlayfs on-disk format. This escaping feature has been supported by overlayfs since Linux 6.7, and allows for you to created images that contain an overlayfs layout inside the image (nested to arbitrary levels).
  • If an image contains trusted.overlay.* xattrs, UnpackLayer will rewrite the xattrs to instead be in the trusted.overlay.overlay.* namespace, so that when merged using overlayfs the user will see the expected xattrs.
  • If an on-disk overlayfs directory used with Generate(Insert)Layer contains escaped trusted.overlay.overlay.* xattrs, they will be rewritten so that the generated layer contains trusted.overlay.* xattrs. If we encounter an unescaped trusted.overlay.* xattr they will not be included in the image (though they may cause the file to be converted to a whiteout in the image) because they are considered to be an internal aspect of the host on-disk format (i.e. trusted.overlay.origin might be automatically set by whatever tool is using the overlayfs layers). Note that in the regular extraction mode, these xattrs will be treated like any other xattrs (this is in contrast to the previous behaviour where they would be silently ignored regardless of the on-disk format being used).
• When extracting a layer, umoci unpack would previously return an error if a tar entry was within a non-directory. In practice such cases are quite unlikely (as layer diffs would usually include an entry changing the type of the non-directory parent) but this could result in spurious errors with somewhat non-standard tar archive layers. Now, umoci will remove the offending non-directory parent component and re-create the parent path as a proper directory tree.
  • This also has the side-effect of fixing the behaviour when unpacking whiteouts with the OverlayfsRootfs on-disk format. If there is a plain whiteout of a regular directory, followed by parent components being made underneath that directory, then the directory should be converted to an opaque whiteout. This matches the behaviour of overlayfs (though again, it seems unlikely that a layer diff tool would generate such a layer). opencontainers/umoci#546

Thanks to all of the following contributors for making this release possible:

• AdamKorcz adam@adalogics.com
• Aleksa Sarai cyphar@cyphar.com
• Bibhas github@bibhasdn.com
• Cameron Nemo cnemo@tutanota.com
• Michael McCracken mikmccra@cisco.com
• Ramkumar Chinchani rchincha@cisco.com
• Serge Hallyn serge@hallyn.com
• Shengjing Zhu i@zhsj.me
• Tycho Andersen tycho@tycho.pizza
• guoguangwu guoguangwu@magic-shield.com

Signed-off-by: Aleksa Sarai cyphar@cyphar.com