This is a long-awaited release of umoci containing some Go API breaking changes, some new features, and many other minor changes and improvements.

Note that the Go API is still considered to be unstable, so downstream users should generally be aware that future updates may contain more breaking changes until we release umoci v1.0.0. However, the umoci CLI is considered to be stable (as it has been widely used for nearly a decade now) and we will endeavour to not make breaking changes.

This version of umoci requires Go 1.23 to build.

Security

• A security flaw was found in the OCI image-spec, where it is possible to cause a blob with one media-type to be interpreted as a different media-type. As umoci is not a registry nor does it handle signatures, this vulnerability had no real impact on umoci but for safety we implemented the now-recommended media-type embedding and verification. CVE-2021-41190

Breaking

• The method of configuring the on-disk format and MapOptions in RepackOptions and UnpackOptions has been changed. The on-disk format is now represented with the OnDiskFormat interface, with DirRootfs and OverlayfsRootfs as possible options to use. MapOptions is now configured inside the OnDiskFormat setting, which will require callers to adjust their usage of the main umoci APIs. In particular, examples like

  unpackOptions := &layer.UnpackOptions{
      MapOptions: mapOptions,
      WhiteoutMode: layer.StandardOCIWhiteout, // or layer.OverlayFSWhiteout
  }
  err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)
  

  will have to now be written as

  unpackOptions := &layer.UnpackOptions{
      OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
          MapOptions: mapOptions,
      },
  }
  err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)
  

  and similarly

  repackOptions := &layer.RepackOptions{
      MapOptions: mapOptions,
      TranslateOverlayWhiteouts: false, // or true
  }
  layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)
  

  will have to now be written as

  repackOptions := &layer.RepackOptions{
      OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
          MapOptions: mapOptions,
      },
  }
  layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)
  

  Note that this means you can easily re-use the OnDiskFormat configuration between both UnpackOptions and RepackOptions, removing the previous need to translate between WhiteoutMode and TranslateOverlayWhiteouts.

  For users of the API that need to extract the MapOptions from UnpackOptions and RepackOptions, there is a new helper MapOptions which will help extract it without doing interface type switching. For OnDiskFormat there is also a Map method that gives you the inner MapOptions regardless of type.

• layer.NewTarExtractor now takes *UnpackOptions rather than UnpackOptions to match the signatures of the other layer.* APIs. Passing nil is equivalent to passing &UnpackOptions{}.

• In umoci 0.4.7, we added support for overlayfs unpacking using the still-unstable Go API. However, the implementation is still missing some key features and so we will now return errors from APIs that are still missing key features:

  • layer.UnpackManifest and layer.UnpackRootfs will now return an error if UnpackOptions.OnDiskFormat is set to anything other than DirRootfs (the default, equivalent to WhiteoutMode being set to OCIStandardWhiteout in umoci 0.4.7).

    This is because bundle-based unpacking currently tries to unpack all layers into the same rootfs and generate an mtree manifest -- this doesn't make sense for overlayfs-style unpacking and will produce garbage bundles as a result. As such, we expect that nobody actually made use of this feature (otherwise we would've seen bug reports complaining about it being completely broken in the past 4 years). opencontainers/umoci#574 tracks re-enabling this feature (and exposing to umoci CLI users, if possible).

    Note that layer.UnpackLayer still supports OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7).

  • Already-extracted bundles with OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7) will now return an error when umoci operates on them -- we included the whiteout mode in our umoci.json but as the feature is broken, umoci will now refuse to operate on such bundles. Such bundles could only have been created using the now-error-inducing UnpackRootfs and UnpackManifest APIs mentioned above, and as mentioned above we expect there to have been no real users of this feature.

Added

• umoci unpack now supports handling layers compressed with zstd. This is something that was added in image-spec v1.2 (which we do not yet support fully) but at least this will allow users to operate on zstd-compressed images, which are slowly becoming more common.
• umoci repack and umoci insert now support creating zstd-compressed layers. The default behaviour (called auto) is to try to match the last layer's compression algorithm, with a fallback to gzip if none of the layer algorithms were supported.
  • Users can specify their preferred compression algorithm using the new --compress flag. You can also disable compression entirely using --compress=none but --compress=auto will never automatically choose none compression.
• GenerateLayer and GenerateInsertLayer with OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci 0.4.7) now support converting trusted.overlay.opaque=y and trusted.overlay.whiteout whiteouts into OCI whiteouts when generating OCI layers.
• OverlayfsRootfs now supports compatibility with the userxattr mount option for overlayfs (where user.overlay.* xattrs are used rather than the default trusted.overlay.*). This is a pretty key compatibility feature for users that use unprivileged overlayfs mounts and will hopefully remove the need for most downstream forks hacking in this functionality (such as stacker). For Go API users, to enable this just set UserXattr: true in OverlayfsRootfs. Note that (as with upstream overlayfs), only one xattr namespace is ever used (so if OverlayfsRootfs.UserXattr == true then trusted.overlay.* xattrs will be treated like any other non-overlayfs xattr).

Changes

• In this release, the primary development branch was renamed to main.
• The runtime-spec version of the config.json version we generate is no longer hard-coded to 1.0.0. We now use the version of the spec we have imported (with any -dev suffix stripped, as such a prefix causes havoc with verification tools -- ideally we would only ever use released versions of the spec but that's not always possible). #452
• Add the cgroup namespace to the default configuration generated by umoci unpack to make sure that our configuration plays nicely with runc when on cgroupv2 systems.
• umoci has been migrated away from github.com/pkg/errors to Go stdlib error wrapping.
• The gzip compression block size has been updated to be more friendly with Docker and other tools that might round-trip the layer blob data (causing the hash to change if the block size is different). #509

Fixed

• In 0.4.7, a performance regression was introduced as part of the VerifiedReadCloser hardening work (to read all trailing bytes) which would cause walk operations on images to hash every blob in the image (even blobs which we couldn't parse and thus couldn't recurse into). To resolve this, we no longer recurse into unparseable blobs. #373 #375 #394
• Handle EINTR on io.Copy operations. Newer Go versions have added more opportunistic pre-emption which can cause EINTR errors in io paths that didn't occur before. #437
• Quite a few changes were made to CI to try to avoid issues with fragility. #452
• umoci will now return an explicit error if you pass invalid uid or gid values to --uid-map and --gid-map rather than silently truncating the value.
• For Go users of umoci, GenerateLayer (but not GenerateInsertLayer) with OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci 0.4.7) had several severe bugs that made the feature unusable:
  • All OCI whiteouts added to the archive would incorrectly have the full host name of the path rather than the correctly rooted path, making the whiteout practically useless.
  • Any non-whiteout files would not be included in the layer, making the layer data incomplete and thus resulting in silent data loss. Given how severe these bugs were and the lack of bug reports of this issue in the past 4 years, it seems this feature has not really been used by anyone (I hope...).
• For Go users of umoci, UnpackLayer now correctly handles several aspects of OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7) extraction that weren't handled correctly:
  • Unlike regular extractions, overlayfs-style extractions require us to create the parent directory of the whiteout (rather than ignoring or assuming the underlying path exists) because the whiteout is being created in a separate layer to the underlying file. We also need to make sure that opaque whiteout targets are directories.
  • trusted.overlay.opaque=y has very peculiar behaviour when a regular whiteout (i.e. mknod c 0 0) is placed inside an opaque directory -- the whiteout-ed file appears in readdir but the file itself doesn't exist. To avoid this confusion (and possible information leak), umoci will no longer extract plain whiteouts within an opaque whiteout directory in the same layer. (As per the OCI spec requirements, this is regardless of the order of the opaque whiteout and the regular whiteout in the layer archive.)

Thanks to all of the following contributors for making this release possible:

• AdamKorcz adam@adalogics.com
• Aleksa Sarai cyphar@cyphar.com
• Bibhas github@bibhasdn.com
• Cameron Nemo cnemo@tutanota.com
• Michael McCracken mikmccra@cisco.com
• Ramkumar Chinchani rchincha@cisco.com
• Serge Hallyn serge@hallyn.com
• Shengjing Zhu i@zhsj.me
• Tycho Andersen tycho@tycho.pizza
• guoguangwu guoguangwu@magic-shield.com

Signed-off-by: Aleksa Sarai cyphar@cyphar.com