Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sparse Zones Unable To Install ooce Packages #138

Open
justledbetter opened this issue May 8, 2023 · 15 comments
Open

Sparse Zones Unable To Install ooce Packages #138

justledbetter opened this issue May 8, 2023 · 15 comments

Comments

@justledbetter
Copy link

(Apologies if I'm opening this in the wrong place again, please feel free to direct me elsewhere if this is a problem further down the stack.)

After upgrading to 151046, I'm no longer able to install local (ooce) packages in sparse zones. The following testcase should reproduce on your end if you're also on 46 (adjusting for your local IP and volume layout):

$ pfexec zadm create -b sparse test < _EOF
{
   "autoboot" : "false",
   "bootargs" : "",
   "brand" : "sparse",
   "fs-allowed" : "",
   "hostid" : "",
   "ip-type" : "exclusive",
   "limitpriv" : "default",
   "net" : [
      {
         "allowed-address" : "10.0.0.19/24",
         "global-nic" : "e1000g0",
         "defrouter": "10.0.0.1",
         "physical" : "test0"
      }
   ],
   "pool" : "",
   "resolvers" : [
      "10.0.0.1"
   ],
   "scheduling-class" : "",
   "zonename" : "test",
   "zonepath" : "/zones/test"
}
_EOF
A ZFS file system has been created for this zone.

       Image: Preparing at /vm/test/root.
Sanity Check: Looking for 'entire' incorporation.
   Publisher: Using omnios (https://pkg.omnios.org/r151046/core/).
   Publisher: Using extra.omnios (https://pkg.omnios.org/r151046/extra/).
       Cache: Using /var/pkg/publisher.
  Installing: Packages (output follows)
Packages to install: 184
Mediators to change:   8
 Services to change:   7

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            184/184     1468/1468      4.7/4.7      --

PHASE                                          ITEMS
Installing new actions                     4996/4996
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 
 Postinstall: Copying SMF seed repository ... done.
        Done: Installation completed in 69.943 seconds.

$ pfexec zadm start test
$ pfexec zlogin test
[Connected to zone 'test' pts/3]
OmniOS r151046  omnios-r151046-82ebda23c9       May 2023
root@test:~# pkg install -v ooce/developer/subversion
Creating Plan (Evaluating mediators): /
pkg install: The files listed below match exclusions which are configured
on this image and can therefore not be installed:

    usr/lib/amd64/libltdl.so
    usr/lib/amd64/libltdl.so.7
    usr/lib/amd64/libltdl.so.7.3.1
    usr/lib/libltdl.so
    usr/lib/libltdl.so.7
    usr/lib/libltdl.so.7.3.1
(...)
See the 'exclude-patterns' and 'exclude-policy' image properties in pkg(1) for
more information.

Is this a problem with my system configuration?

Is there anything I can do to help you debug this further?

Is this something that needs to be raised on an OmniOSce or Illumos bug system?

Any input you can provide would be greatly appreciated!

@moetiker
Copy link
Member

moetiker commented May 8, 2023

I would use fs option, so you do not have to install any ooce ... in the spacezone zone

  "fs" : [
      {
         "dir" : "/opt/ooce",
         "options" : [
            "ro"
         ],
         "special" : "/opt/ooce",
         "type" : "lofs"
      }
   ],

@justledbetter
Copy link
Author

I can certainly try that - seems simple enough, but this means I'd need to install any packages I want to use in a sparse zone on the main server itself, no?

Is there no other way to do this? It used to work okay, at least a couple of releases ago.

@citrus-it
Copy link
Member

root@test:~# pkg install -v ooce/developer/subversion
Creating Plan (Evaluating mediators): /
pkg install: The files listed below match exclusions which are configured
on this image and can therefore not be installed:

usr/lib/amd64/libltdl.so
usr/lib/amd64/libltdl.so.7
usr/lib/amd64/libltdl.so.7.3.1
usr/lib/libltdl.so
usr/lib/libltdl.so.7
usr/lib/libltdl.so.7.3.1

(...)
See the 'exclude-patterns' and 'exclude-policy' image properties in pkg(1) for
more information.


Is this a problem with my system configuration?

Is there anything I can do to help you debug this further?

Is this something that needs to be raised on an OmniOSce or Illumos bug system?

Any input you can provide would be greatly appreciated!

What's happening here is that the ooce/developer/subversion package has a dependency which is pulling in other packages, at least one of which delivers files into /usr and won't work in a sparse zone. Any such dependent packages will need to be installed in the GZ..

Looking at the files that were shown in the output:

% pkg search libltdl.so
INDEX      ACTION VALUE                    PACKAGE
basename   link   usr/lib/amd64/libltdl.so pkg:/library/libtool/[email protected]
basename   link   usr/lib/libltdl.so       pkg:/library/libtool/[email protected]
basename   link   usr/lib/amd64/libltdl.so pkg:/library/libtool/[email protected]
basename   link   usr/lib/libltdl.so       pkg:/library/libtool/[email protected]

So in this case, I would install library/libtool/libltdl in the GZ, recursively so it ends up in the sparse zone too, and then you should be able to install subversion in there.

GZ# pkg install -r libltdl
NGZ# pkg install subversion

@justledbetter
Copy link
Author

When I query the global zone, it's showing as being installed already:

$ pkg list | grep libltdl
library/libtool/libltdl                           2.4.6-151046.0             i--

But when I try running install -r, it bombs out too:

$ pfexec pkg install -r libltdl
No updates necessary for this image.
Planning linked: 0/2 done; 1 working: zone:test
Linked progress: \pkg: install failed (linked image exception(s)):

A 'install' operation failed for child 'zone:test' with an unexpected
return value of 1 and generated the following output:

pkg install: The files listed below match exclusions which are configured
on this image and can therefore not be installed:

    usr/lib/amd64/libltdl.so
    usr/lib/amd64/libltdl.so.7
    usr/lib/amd64/libltdl.so.7.3.1
    usr/lib/libltdl.so
    usr/lib/libltdl.so.7
    usr/lib/libltdl.so.7.3.1

See the 'exclude-patterns' and 'exclude-policy' image properties in pkg(1) for
more information.

Note these files do show up in the child zone:

# ls -lh /usr/lib/libltdl* /usr/lib/amd64/libltdl*
lrwxrwxrwx   1 root     root          16 Feb 10 01:20 /usr/lib/amd64/libltdl.so -> libltdl.so.7.3.1
lrwxrwxrwx   1 root     root          16 Feb 10 01:20 /usr/lib/amd64/libltdl.so.7 -> libltdl.so.7.3.1
-rwxr-xr-x   1 root     bin        55.9K May  4 12:52 /usr/lib/amd64/libltdl.so.7.3.1
lrwxrwxrwx   1 root     root          16 Feb 10 01:20 /usr/lib/libltdl.so -> libltdl.so.7.3.1
lrwxrwxrwx   1 root     root          16 Feb 10 01:20 /usr/lib/libltdl.so.7 -> libltdl.so.7.3.1
-rwxr-xr-x   1 root     bin        46.4K May  4 12:52 /usr/lib/libltdl.so.7.3.1

@citrus-it
Copy link
Member

In the NGZ, try changing the exclude-policy value. It looks like you've found a bug that we'll need to fix, but this should hopefully get things working.

NGZ# pkg set exclude-policy warn

and then try the recursive installation in the GZ again.

@justledbetter
Copy link
Author

That does get the package installation working again, thanks!

It does still print out the same list of exclusions, I'll paste here for reference in case you need it.

test# pkg set-property exclude-policy warn
test# pkg install ooce/developer/subversion
           Packages to install: 13
           Mediators to change:  2
            Services to change:  3
       Create boot environment: No
Create backup boot environment: No

*****************************************************************************
WARNING: The following actions have not been installed as this is a partial
image (there are configured exclusions):

    usr/bin/amd64/dltest
    usr/bin/amd64/isql
    usr/bin/amd64/iusql
    usr/bin/amd64/odbc_config
    usr/bin/amd64/odbcinst
    usr/bin/amd64/slencheck
    usr/bin/dltest
    usr/bin/i386
    usr/bin/i386/dltest
    usr/bin/i386/isql
    usr/bin/i386/iusql
    usr/bin/i386/odbc_config
    usr/bin/i386/odbcinst
    usr/bin/i386/slencheck
    usr/bin/isql
    usr/bin/iusql
    usr/bin/odbc_config
    usr/bin/odbcinst
    usr/bin/slencheck
    usr/include
    usr/include/odbc
    usr/include/odbc/autotest.h
    usr/include/odbc/odbcinst.h
    usr/include/odbc/odbcinstext.h
    usr/include/odbc/sql.h
    usr/include/odbc/sqlext.h
    usr/include/odbc/sqlspi.h
    usr/include/odbc/sqltypes.h
    usr/include/odbc/sqlucode.h
    usr/include/odbc/unixODBC
    usr/include/odbc/unixODBC/config.h
    usr/include/odbc/unixODBC/unixodbc_conf.h
    usr/include/odbc/unixodbc.h
    usr/include/odbc/uodbc_extras.h
    usr/include/odbc/uodbc_stats.h
    usr/lib/amd64/libnn.so
    usr/lib/amd64/libnn.so.1
    usr/lib/amd64/libnn.so.1.0.0
    usr/lib/amd64/libodbc.so
    usr/lib/amd64/libodbc.so.2
    usr/lib/amd64/libodbc.so.2.0.0
    usr/lib/amd64/libodbccr.so
    usr/lib/amd64/libodbccr.so.2
    usr/lib/amd64/libodbccr.so.2.0.0
    usr/lib/amd64/libodbcinst.so
    usr/lib/amd64/libodbcinst.so.2
    usr/lib/amd64/libodbcinst.so.2.0.0
    usr/lib/amd64/libodbcpsql.so
    usr/lib/amd64/libodbcpsql.so.2
    usr/lib/amd64/libodbcpsql.so.2.0.0
    usr/lib/amd64/libtemplate.so
    usr/lib/amd64/libtemplate.so.1
    usr/lib/amd64/libtemplate.so.1.0.0
    usr/lib/amd64/pkgconfig
    usr/lib/amd64/pkgconfig/odbc.pc
    usr/lib/amd64/pkgconfig/odbccr.pc
    usr/lib/amd64/pkgconfig/odbcinst.pc
    usr/lib/libnn.so
    usr/lib/libnn.so.1
    usr/lib/libnn.so.1.0.0
    usr/lib/libodbc.so
    usr/lib/libodbc.so.2
    usr/lib/libodbc.so.2.0.0
    usr/lib/libodbccr.so
    usr/lib/libodbccr.so.2
    usr/lib/libodbccr.so.2.0.0
    usr/lib/libodbcinst.so
    usr/lib/libodbcinst.so.2
    usr/lib/libodbcinst.so.2.0.0
    usr/lib/libodbcpsql.so
    usr/lib/libodbcpsql.so.2
    usr/lib/libodbcpsql.so.2.0.0
    usr/lib/libtemplate.so
    usr/lib/libtemplate.so.1
    usr/lib/libtemplate.so.1.0.0
    usr/lib/pkgconfig
    usr/lib/pkgconfig/odbc.pc
    usr/lib/pkgconfig/odbccr.pc
    usr/lib/pkgconfig/odbcinst.pc
    usr/share/man/man1/dltest.1
    usr/share/man/man1/isql.1
    usr/share/man/man1/iusql.1
    usr/share/man/man1/odbc_config.1
    usr/share/man/man1/odbcinst.1
    usr/share/man/man5/odbc.ini.5
    usr/share/man/man5/odbcinst.ini.5
    usr/share/man/man7
    usr/share/man/man7/unixODBC.7
    usr/lib/amd64/libltdl.so
    usr/lib/amd64/libltdl.so.7
    usr/lib/amd64/libltdl.so.7.3.1
    usr/lib/libltdl.so
    usr/lib/libltdl.so.7
    usr/lib/libltdl.so.7.3.1
    usr/bin/amd64/xslt-config
    usr/bin/amd64/xsltproc
    usr/bin/i386
    usr/bin/i386/xslt-config
    usr/bin/i386/xsltproc
    usr/bin/xslt-config
    usr/bin/xsltproc
    usr/include
    usr/include/libexslt
    usr/include/libexslt/exslt.h
    usr/include/libexslt/exsltconfig.h
    usr/include/libexslt/exsltexports.h
    usr/include/libxslt
    usr/include/libxslt/attributes.h
    usr/include/libxslt/documents.h
    usr/include/libxslt/extensions.h
    usr/include/libxslt/extra.h
    usr/include/libxslt/functions.h
    usr/include/libxslt/imports.h
    usr/include/libxslt/keys.h
    usr/include/libxslt/namespaces.h
    usr/include/libxslt/numbersInternals.h
    usr/include/libxslt/pattern.h
    usr/include/libxslt/preproc.h
    usr/include/libxslt/security.h
    usr/include/libxslt/templates.h
    usr/include/libxslt/transform.h
    usr/include/libxslt/variables.h
    usr/include/libxslt/xslt.h
    usr/include/libxslt/xsltInternals.h
    usr/include/libxslt/xsltconfig.h
    usr/include/libxslt/xsltexports.h
    usr/include/libxslt/xsltlocale.h
    usr/include/libxslt/xsltutils.h
    usr/lib/amd64/libexslt.so
    usr/lib/amd64/libexslt.so.0
    usr/lib/amd64/libexslt.so.0.8.18
    usr/lib/amd64/libxslt-plugins
    usr/lib/amd64/libxslt.so
    usr/lib/amd64/libxslt.so.1
    usr/lib/amd64/libxslt.so.1.1.30
    usr/lib/amd64/pkgconfig
    usr/lib/amd64/pkgconfig/libexslt.pc
    usr/lib/amd64/pkgconfig/libxslt.pc
    usr/lib/amd64/xsltConf.sh
    usr/lib/libexslt.so
    usr/lib/libexslt.so.0
    usr/lib/libexslt.so.0.8.18
    usr/lib/libxslt-plugins
    usr/lib/libxslt.so
    usr/lib/libxslt.so.1
    usr/lib/libxslt.so.1.1.30
    usr/lib/pkgconfig
    usr/lib/pkgconfig/libexslt.pc
    usr/lib/pkgconfig/libxslt.pc
    usr/lib/xsltConf.sh
    usr/share/aclocal
    usr/share/aclocal/libxslt.m4
    usr/share/man/man1/xsltproc.1
    usr/share/man/man3
    usr/share/man/man3/libexslt.3
    usr/share/man/man3/libxslt.3
*****************************************************************************

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                              13/13     1448/1448    29.3/29.3  1.4M/s

PHASE                                          ITEMS
Installing new actions                     2400/2400
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 
Reading search index                            Done 
Updating search index                          13/13 
Updating package cache                           2/2 

@citrus-it
Copy link
Member

Great, You may find you need to install a few more of the pre-requisite packages in the GZ (to get the files into the shared /usr).

@justledbetter
Copy link
Author

Just wanted to check in on this issue - The long term result of making this change is that instead of updating all of my Zones by updating the global zone and rebooting, I now have to login to each zone and run pkg update as well. This is a lot less optimal than Sparse zones were a couple of years ago... Do we have any idea why this change was made, and whether anything can be done to restore the original functionality?

I suspect the issue relates to the Sparse zone containing incomplete information about what packages IPS has carried into the zone from the Global Zone (via /var/pkg), but I'm not deep enough in the internals of the system to know whether or not this is the case...

@justledbetter
Copy link
Author

Coming back up from Patch Tuesday, I just realized I'm still going through this process a year later - lipkg and sparse zones in my environment do not receive updates from the Global Zone anymore, and each nonglobal zone requires me to zlogin to it specifically and run pkg update -v on it after updating and rebooting the GZ.

Is there any way to debug this issue so we can get it fixed? It's driving me crazy to update dozens of zones every month instead of just updating the hosting computers themselves. :)

@justledbetter justledbetter reopened this Oct 9, 2024
@justledbetter
Copy link
Author

(Note that I've tried per the OmniOS documentation to do the following, and it also does not work - the NGZs still require manual pkg update operations to be applied within themselves afterwards.)

GZ# pkg update -v
(reboot)
GZ# zadm stop NGZ
GZ# zoneadm -z NGZ detach
GZ# zoneadm -z NGZ attach
GZ# zadm start NGZ
GZ# zlogin NGZ
NGZ# pkg update -v
(finds and installs all of the updates applied to the GZ, and creates a new BE backup for the zone)

My expectation and hope is that the original way it worked will eventually work again:

GZ# pkg update -v
(reboot)
(all sparse and lipkg zones are up to date other than OOCE packages that require updates)

@hadfl
Copy link
Member

hadfl commented Oct 9, 2024

The detach, attach mechanism is only needed for ipkg (i.e. non-linked zones). All you have to do with linked/sparse zones is to use the -r option when engaging pkg from the GZ.

From the man page:

           -r      Run this operation in the global zone and also in all
                   installed linked non-global zones.  The effect on the non-
                   global zone is similar to logging into each non-global zone
                   and running the command directly.  Without this option,
                   when you run pkg commands in the global zone, non-global
                   zones are modified only to the extent required to keep them
                   compatible with the global zone.  With this option, the pkg
                   operation is applied to all installed non-global zones
                   except as limited by the -z and -Z options.  Zones that are
                   excluded by the -z and -Z options might still be modified
                   if updates are required to keep them in sync with the
                   global zone.

@citrus-it
Copy link
Member

You can also make this the default behaviour by setting the default-recurse image property.

From the pkg(1) man page:

     default-recurse
             (boolean) If set to True, the package client will default to
             recursing into child images for pkg update operations, as if the
             -r option had been provided on the command line.  This behaviour
             can be overriden on a per-case basis via the -R option.

             Default value: False

@justledbetter
Copy link
Author

I apologize for not seeing your replies sooner - Thank you very much for the pointers! I'll give this a try as soon as I can and report back if things still seem weird! 😁

@justledbetter
Copy link
Author

justledbetter commented Dec 17, 2024

Running pkg update -Rv does not update any packages within child zones, I still have to login to each zone individually to pkg update -v afterwards. (Is this expected? Am I calling it wrong?)

I've tried setting the default-recurse property to True on one of my instances, but that host is slightly ahead of the rest, and so it doesn't have any updates at the moment. Will keep an eye on it to see if that makes a difference next update cycle.

Update: Reading through the documentation and am guessing I did, indeed, call it wrong. Will try again with -r next time (Sorry :) )

@citrus-it
Copy link
Member

Running pkg update -Rv does not update any packages within child zones, I still have to login to each zone individually to pkg update -v afterwards. (Is this expected? Am I calling it wrong?)

Yes, the -R flag disables this. You want -r.

-R      Disable recursion into linked non-global zones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants