Merge pull request #458 from citrus-it/sync

Sync fixes from upstream

README.sync

@@ -4,11 +4,11 @@
 The pkg5 components have been updated to the latest upstream solaris-ips
 as of:
 
-commit bd2027fc9459d3f362d80ecc76092d0cd080ab97
-Author: Chandana R <chandana.r@oracle.com>
-Date:   Thu Feb 2 03:21:24 2023 -0800
+commit c5674fc26b31cd234007b090c451aa4ea0e0a668
+Author: Jakub Kulik <jakub.kulik@oracle.com>
+Date:   Fri Oct 20 06:43:37 2023 -0700
 
-    34872181 pkgrecv MemoryError due to 8GB memory limit
+    34044891 pkg verify is outputting ELF error that makes no sense
 
 -----------------------------------------------------------------------------
 

doc/client_api_versions.txt

@@ -739,7 +739,7 @@ Version 39:
 Compatible with clients using versions 36-38.
 
     Introduces the notion of a system repository to the publisher
-    heirarchy.  This specific type of repository is a subclass of
+    hierarchy.  This specific type of repository is a subclass of
     RepositoryURI.  A few new api exceptions are added, too.
 
 Version 38:

doc/dev-guide/chpt7.txt

@@ -89,21 +89,21 @@ are not installed in non-global zones.  This is typically done in the
 manifest during publication with a |pkgmogrify| rule.  Thus the packages
 from the i386 and sparc builds are already marked for zones.  We then
 use the |pkgmerge| command to take the packages from the sparc and i386
-builds and merge them together.  This is far more reliable and faster 
+builds and merge them together.  This is far more reliable and faster
 than attempting to construct such packages manually.
 
-In general, it is not practical to define new variants without
-modifying the packaging system as no practical means currently exists
-for defining a default value for variants in general.  Package developers
-cannot define new variants at present.
+The ``variant.debug.*`` portion of the variant namespace is explicitly
+predefined to have a default version of ``false``; thus, developers
+can provide debug versions of their components, tagged with the
+appropriate variant, and users can select that variant if problems
+arise.  Remember that variants are set per image, so selecting a
+suitable name that is unique at the appropriate resolution for that
+piece of software is important.
 
-However, the ``variant.debug.*`` portion of the variant namespace is predefined
-to have a default version of ``false``; thus, developers can provide debug
-versions of their components, tagged with the appropriate variant, and
-users can select that variant if problems arise.  Remember that
-variants are set per image, so selecting a suitable name that
-is unique at the appropriate resolution for that piece of software 
-is important.
+In addition, any variant tags not described here are assumed to
+have a default value of ``false`` in the image.  This allows
+the definition of custom variants not explicitly set in the
+image for use in packages.
 
 Note that variant tags are applied to any actions that differ between
 architectures during merging; this includes dependencies, ``set`` actions,
@@ -120,7 +120,7 @@ Facets
 Often, package developers have optional portions of their software
 that actually belong with the main body, but some people might not
 want to install.  Some examples are localization files for different
-locales, man pages and other documentation, header files needed 
+locales, man pages and other documentation, header files needed
 only by developers or DTrace users.
 
 Traditionally, such optional content has been placed in separate
@@ -141,8 +141,8 @@ with matching rules such that the longest match wins.
 
 For example, the default value for all facets is ``true``; the |pkg| client
 implicitly sets ``facet.*`` to ``true``.
-Documentation in Oracle Solaris packages is tagged with the type of 
-documentation.  For example, man pages are tagged with 
+Documentation in Oracle Solaris packages is tagged with the type of
+documentation.  For example, man pages are tagged with
 ``facet.doc.man=true`` in the package manifests.
 
 The following commands include man pages but exclude all other documentation
@@ -244,7 +244,7 @@ facet.locale.en_SG        facet.locale.ku           facet.locale.zh_HK
 facet.locale.en_US        facet.locale.ku_TR        facet.locale.zh_HK
 facet.locale.en_ZW        facet.locale.ky           facet.locale.zh_SG
 facet.locale.eo           facet.locale.ky_KG        facet.locale.zh_TW
-facet.locale.es_AR        facet.locale.lg     
+facet.locale.es_AR        facet.locale.lg
 ========================  ==================        ==================
 
 

doc/facets.txt

@@ -2,13 +2,13 @@ Package facets and variants
 ---------------------------
 
 Traditionally, packaging systems have placed optional components
-of a package in separate packages, and established conventions 
+of a package in separate packages, and established conventions
 for naming such components, such as -localization-locale,
--devel, -doc, etc.  This method of ad-hoc decomposition 
+-devel, -doc, etc.  This method of ad-hoc decomposition
 makes it more difficult for GUI programs to offer the appropriate
 choices when selecting components, makes the introduction of new
 optional components difficult and makes installing documentation
-after the fact a painful process.  
+after the fact a painful process.
 
 Packaging options also exist which are mutually exclusive; the typical
 example is which architecture the package supports.  One cannot select
@@ -46,7 +46,7 @@ if any of the variant tags do not match, the action is not installed.
 On the other hand, the facet tags are OR'd together; if any of the
 facets match the action is installed.
 
-Facets and variants are tags, and as such can initially be 
+Facets and variants are tags, and as such can initially be
 set on any action, including dependencies, etc.  This can make
 testing problematic, however.  To simplify matters, variants and
 facets are set at the image level.  Package developers desiring
@@ -77,11 +77,11 @@ A more useful example would be installing the french locale as
 spoken in France.  This consists of files tagged
 
 facet.locale.fr, which tags files which should be installed for all
-French locales, 
+French locales,
 
 and
 
-facet.locale.fr_FR, which is for France in particular, 
+facet.locale.fr_FR, which is for France in particular,
 
 but not
 
@@ -90,11 +90,11 @@ facet.locale.fr_CA, which is for Canada.
 Setting the following facets insures this selection:
 
 facet.locale.*     false   # install only locales we specify
-facet.locale.fr    true    
+facet.locale.fr    true
 facet.locale.fr_FR true
 
 Changing either variants or facets for an installed image
-effectively causes re-evaluation of the actions in the installed 
+effectively causes re-evaluation of the actions in the installed
 packages, and may or may not be done live depending on the impact
 of the change.
 
@@ -117,9 +117,10 @@ facet.doc.html		 true
 facet.devel		 true
 facet.debug.<all>	 false
 facet.optional.<all>	 false
-facet.platform.sun4u	 true 
+facet.platform.sun4u	 true
 facet.platform.sun4v	 true
 variant.arch		 one of sparc, i386 as set by platform code
 variant.opensolaris.zone either global or nonglobal as set by image type
 variant.debug.<all>	 false
+variant.<unknown>        false   all unknown variants default to 'false'
 

src/depot-config.py

@@ -453,7 +453,7 @@ def _createCertificateKey(
     issuerCert=None,
     issuerKey=None,
     key_type=TYPE_RSA,
-    key_bits=1024,
+    key_bits=2048,
     digest="sha256",
 ):
     """Generate a certificate given a certificate request.

src/man/pkg.7

@@ -1867,7 +1867,7 @@ The exceptions are the
 variants.
 The
 .Sy debug
-variants can only be set to
+variants should only be set to
 .Sy true
 or
 .Sy false ;
@@ -1886,7 +1886,9 @@ file group=sys mode=0644 overlay=allow owner=root \e
 .Ed
 .Pp
 The variant value must be set on the image in order for a package using the
-variant to be installed.
+variant to be installed; by default all unspecified variants have the value
+.Sy false ,
+which may or may not make the package installable.
 The
 .Sy arch
 and

src/modules/_varcet.c

@@ -229,10 +229,10 @@ _allow_variant(PyObject *self, PyObject *args, PyObject *kwargs)
 			if (sysv == NULL) {
 				/*
 				 * If system variant value doesn't exist, then
-				 * allow the action if it is a debug variant
-				 * that is "false".
+				 * allow the action if it is a variant that is
+				 * "false".
 				 */
-				if ((strncmp(as, "variant.debug.", 14) == 0) &&
+				if ((strncmp(as, "variant.", 8) == 0) &&
 				    (strncmp(av, "false", 5) != 0)) {
 					Py_DECREF(act_attrs);
 					Py_RETURN_FALSE;

src/modules/actions/group.py

@@ -21,7 +21,7 @@
 #
 
 #
-# Copyright (c) 2008, 2018, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2008, 2023, Oracle and/or its affiliates.
 #
 
 """module describing a user packaging object
@@ -196,9 +196,18 @@ def verify(self, img, **args):
 
         should_be = grdefval.copy()
         should_be.update(self.attrs)
-        # Note where attributes are missing
-        for k in should_be:
-            cur_attrs.setdefault(k, "<missing>")
+
+        # Ignore optional package metadata
+        skip_list = ["variant", "facet"]
+
+        # Note where attributes are missing, but remove those that are
+        # in the skip list
+        for k in list(should_be):
+            if any(token in k for token in skip_list):
+                should_be.pop(k, None)
+            else:
+                cur_attrs.setdefault(k, "<missing>")
+
         # Note where attributes should be empty
         for k in cur_attrs:
             if cur_attrs[k]:

src/modules/actions/user.py

@@ -21,7 +21,7 @@
 #
 
 #
-# Copyright (c) 2008, 2016, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2008, 2023, Oracle and/or its affiliates.
 #
 
 """module describing a user packaging object
@@ -289,7 +289,7 @@ def verify(self, img, **args):
 
         # ignore changes in certain fields if password is
         # mutable; this indicates that this account is used
-        # by a human and logins, timeouts, etc. are changable.
+        # by a human and logins, timeouts, etc. are changeable.
         if should_be["password"] in self.mutable_passwords:
             for attr in self.use_existing_attrs:
                 if attr in should_be:
@@ -306,9 +306,15 @@ def verify(self, img, **args):
         # always ignore flag
         if "flag" in cur_attrs:
             del cur_attrs["flag"]
-        # Note where attributes are missing
-        for k in should_be:
-            cur_attrs.setdefault(k, "<missing>")
+
+        # Ignore optional package metadata and flags
+        skip_list = ["variant", "facet", "flag"]
+        for k in list(should_be):
+            if any(token in k for token in skip_list):
+                should_be.pop(k, None)
+            else:
+                cur_attrs.setdefault(k, "<missing>")
+
         # Note where attributes should be empty
         for k in cur_attrs:
             if cur_attrs[k]:

src/modules/client/api.py

@@ -1185,8 +1185,6 @@ def gen_variants(self, variant_list, implicit=False, patterns=misc.EmptyI):
                 for p in patterns
                 if "*" not in p and "?" not in p
             )
-            # Only debug variants can have an implicit value.
-            iset = set(p for p in iset if p.startswith("variant.debug."))
         vlist = sorted(vimg | set(vpkg.keys()) | iset)
 
         # Generate the results.
@@ -5301,7 +5299,7 @@ def __set_can_be_canceled(self, status):
             self.__cancel_state_callable(self.__can_be_canceled)
 
     def reset(self):
-        """Resets the API back the initial state. Note:
+        """Resets the API back to the initial state. Note:
         this does not necessarily return the disk to its initial state
         since the indexes or download cache may have been changed by
         the prepare method."""

src/modules/elfextract.c

@@ -20,7 +20,7 @@
  */
 
 /*
- *  Copyright (c) 2009, 2017, Oracle and/or its affiliates. All rights reserved.
+ *  Copyright (c) 2009, 2023, Oracle and/or its affiliates.
  *  Copyright 2018 OmniOS Community Edition (OmniOSce) Association.
  */
 
@@ -684,7 +684,8 @@ gethashes(int fd, int doelf, int do256, int do512)
 	}
 
 	if (elf_kind(elf) != ELF_K_ELF || !elf_getshstrndx(elf, &sh_str)) {
-		PyErr_SetString(ElfError, elf_errmsg(-1));
+		/* No error is set at this point. */
+		PyErr_SetString(ElfError, "invalid file type");
 		goto err;
 	}
 

src/modules/misc.py

@@ -469,11 +469,11 @@ def gunzip_from_stream(
         shasum = hash_func()
     dcobj = zlib.decompressobj(-zlib.MAX_WBITS)
 
-    def writeout(buf):
+    def writeout(buffer):
         if isinstance(outfile, StringIO):
-            outfile.write(ubuf.decode())
+            outfile.write(buffer.decode(errors="replace"))
         else:
-            outfile.write(ubuf)
+            outfile.write(buffer)
 
     while True:
         buf = gz.read(64 * 1024)

src/modules/server/depot.py

@@ -1573,7 +1573,7 @@ def __get_matching_p5i_data(self, rstore, pfmri):
         if not matches:
             return ""
 
-        if not "@" in pfmri or "*" in pfmri:
+        if "@" not in pfmri or "*" in pfmri:
             # When using wildcards or exact name match, trim the
             # results to only the unique package stems.
             matches = sorted(set([m.pkg_name for m in matches]))
@@ -1665,7 +1665,7 @@ def status_0(self, *tokens):
             )
         except Exception as e:
             raise cherrypy.HTTPError(
-                http_client.NOT_FOUND, _("Unable " "to generate statistics.")
+                http_client.NOT_FOUND, _("Unable to generate statistics.")
             )
         return misc.force_bytes(out + "\n")
 
@@ -1698,10 +1698,10 @@ def nasty_before_handler(nasty_depot, maxroll=100):
             response = cherrypy.response
             response.body = random.choice(
                 [
-                    "",
-                    "set this is a bogus action",
-                    "Instead of office chair, " "package contained bobcat.",
-                    '{"this is a": "fragment of json"}',
+                    b"",
+                    b"set this is a bogus action",
+                    b"Instead of office chair, package contained bobcat.",
+                    b'{"this is a": "fragment of json"}',
                 ]
             )
             return True
@@ -2179,7 +2179,7 @@ def nasty_serve_file(self, filepath, content_type):
             # NASTY
             # Write garbage into the response
             cherrypy.log("NASTY serve_file: prepend garbage")
-            response.body = "NASTY!"
+            response.body = b"NASTY!"
             response.body += nfile.read(filesz)
             # If we're sending data, lie about the length and
             # make the client catch us.
@@ -2206,6 +2206,7 @@ def nasty_serve_file(self, filepath, content_type):
         else:
             response.body = nfile.read(filesz)
 
+        nfile.close()
         return response.body
 
 

src/modules/server/transaction.py

@@ -21,7 +21,7 @@
 #
 
 #
-# Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2007, 2023, Oracle and/or its affiliates.
 #
 
 from __future__ import print_function
@@ -706,6 +706,9 @@ def add_manifest(self, f):
 
         if isinstance(f, six.string_types):
             f = open(f, "rb")
+            opened = True
+        else:
+            opened = False
         # Store the manifest file.
         fpath = os.path.join(self.dir, "manifest")
         with open(fpath, "ab+") as wf:
@@ -719,6 +722,9 @@ def add_manifest(self, f):
                 f.seek(0)
                 content = f.read()
                 wf.write(content)
+            finally:
+                if opened:
+                    f.close()
         # Do some sanity checking on packages marked or being marked
         # obsolete or renamed.
         m = pkg.manifest.Manifest()