Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LTFS Mount Failure on iSCSI Tape Drive: XML Parser and I/O Errors (-21716) #484

Open
saltarob opened this issue Oct 6, 2024 · 11 comments
Open

LTFS Mount Failure on iSCSI Tape Drive: XML Parser and I/O Errors (-21716) #484

saltarob opened this issue Oct 6, 2024 · 11 comments
Labels
question

Comments

@saltarob
Copy link

saltarob commented Oct 6, 2024

I am experiencing issues when attempting to mount a tape formatted with LTFS on an iSCSI tape drive. After successfully connecting to the iSCSI drive and formatting the tape, I encounter the following errors when trying to mount the volume:

7395 LTFS12049E Cannot read: backend call failed (-21716).
7395 LTFS17039E XML parser: failed to read a block from the medium (-21716).
7395 LTFS17037E XML parser: failed to read from XML stream.
7395 LTFS17273E XML parser: Detected an I/O error on reading an index from the tape (-21716).
7395 LTFS17016E Cannot parse index direct from medium (-21716).
7395 LTFS11194W Cannot read index: failed to read and parse XML data (-21716).
7395 LTFS11024E Cannot mount volume: read index failed on the index partition.
7395 LTFS14013E Cannot mount the volume from device.
7395 LTFS30252I Logical block protection is disabled.

Could you please confirm if LTFS officially supports iSCSI drives? Any insights into the errors I’m encountering or general guidance on using LTFS with iSCSI would be greatly appreciated.

Command Line Output

Here are the command outputs I executed before encountering the issue:

  1. Listing the devices:
[user@almalinux-vm ~]$ ltfs -o device_list
7392 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
7392 LTFS14058I LTFS Format Specification version 2.4.0.
7392 LTFS14104I Launched by "ltfs -o device_list".
7392 LTFS14105I This binary is built for Linux (x86_64).
7392 LTFS14106I GCC version is 8.5.0 20210514 (Red Hat 8.5.0-22).
7392 LTFS17087I Kernel version: Linux version 4.18.0-553.22.1.el8_10.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #1 SMP Tue Sep 24 05:16:59 EDT 2024 i386.
7392 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7392 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7392 LTFS17089I Distribution: NAME="AlmaLinux".
7392 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7392 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7392 LTFS17085I Plugin: Loading "sg" tape backend.
Tape Device list:
Device Name = /dev/sg1 (6.0.0.0), Vendor ID = QUANTUM, Product ID = ULTRIUM-HH7, Serial Number = 90WT021621, Product Name =[ULTRIUM-HH7].
  1. Formatting the tape:
[user@almalinux-vm ~]$ mkltfs -d /dev/sg1
LTFS15000I Starting mkltfs, LTFS version 2.5.0.0 (Prelim), log level 2.
LTFS15041I Launched by "mkltfs -d /dev/sg1".
LTFS15042I This binary is built for Linux (x86_64).
LTFS15043I GCC version is 8.5.0 20210514 (Red Hat 8.5.0-22).
LTFS17087I Kernel version: Linux version 4.18.0-553.22.1.el8_10.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #1 SMP Tue Sep 24 05:16:59 EDT 2024 i386.
LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
LTFS17089I Distribution: NAME="AlmaLinux".
LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
LTFS15003I Formatting device '/dev/sg1'.
LTFS15004I LTFS volume blocksize: 524288.
LTFS15005I Index partition placement policy: None.
LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x5626409f3bf0).
LTFS17085I Plugin: Loading "sg" tape backend.
LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg1).
LTFS30250I Opened the SCSI tape device 6.0.0.0 (/dev/sg1).
LTFS30207I Vendor ID is QUANTUM.
LTFS30208I Product ID is ULTRIUM-HH7.
LTFS30214I Firmware revision is MA71.
LTFS30215I Drive serial is 90WT021621.
LTFS30285I The reserved buffer size of /dev/sg1 is 1048576.
LTFS30294I Setting up timeout values from RSOC.
LTFS17160I Maximum device block size is 1048576.
LTFS11330I Loading cartridge.
LTFS11332I Load successful.
LTFS17157I Changing the drive setting to write-anywhere mode.
LTFS15049I Checking the medium (mount).
LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 10.
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS15010I Creating data partition b on SCSI partition 1.
LTFS15011I Creating index partition a on SCSI partition 0.
LTFS17165I Resetting the medium's capacity proportion.
LTFS11097I Partitioning the medium.
LTFS30263I MODESELECT returns Mode Parameters Rounded (-20101) /dev/sg1.
LTFS11100I Writing label to partition b.
LTFS11278I Writing index to partition b.
LTFS30205I READ_ATTR (0x8c) returns -20501.
LTFS30263I READ_ATTR returns Invalid Field in CDB (-20501) /dev/sg1.
LTFS30233I Cannot read attribute (-20501).
LTFS11336I The attribute does not exist. Ignore the expected error.
LTFS17235I Writing index of NO_BARCODE to b (Reason: Format, 0 files) 90WT021621.
LTFS17236I Wrote index of NO_BARCODE (Gen = 1, Part = b, Pos = 5, 90WT021621).
LTFS11337I Update index-dirty flag (0) - NO_BARCODE (0x0x5626409f3bf0).
LTFS11100I Writing label to partition a.
LTFS11278I Writing index to partition a.
LTFS30205I READ_ATTR (0x8c) returns -20501.
LTFS30263I READ_ATTR returns Invalid Field in CDB (-20501) /dev/sg1.
LTFS30233I Cannot read attribute (-20501).
LTFS11336I The attribute does not exist. Ignore the expected error.
LTFS17235I Writing index of NO_BARCODE to a (Reason: Format, 0 files) 90WT021621.
LTFS17236I Wrote index of NO_BARCODE (Gen = 1, Part = a, Pos = 5, 90WT021621).
LTFS15013I Volume UUID is: 0d901583-1f98-4571-8153-b7d8000d3fd2.
LTFS15019I Volume capacity is 5732 GB.
LTFS30252I Logical block protection is disabled.
LTFS15024I Medium formatted successfully.
  1. Attempting to mount the volume:
[user@almalinux-vm ~]$ ltfs -o devname=/dev/sg1 /ltfs
7395 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
7395 LTFS14058I LTFS Format Specification version 2.4.0.
7395 LTFS14104I Launched by "ltfs -o devname=/dev/sg1 /ltfs".
7395 LTFS14105I This binary is built for Linux (x86_64).
7395 LTFS14106I GCC version is 8.5.0 20210514 (Red Hat 8.5.0-22).
7395 LTFS17087I Kernel version: Linux version 4.18.0-553.22.1.el8_10.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #1 SMP Tue Sep 24 05:16:59 EDT 2024 i386.
7395 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7395 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7395 LTFS17089I Distribution: NAME="AlmaLinux".
7395 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7395 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7395 LTFS17085I Plugin: Loading "sg" tape backend.
LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg1).
LTFS30250I Opened the SCSI tape device 6.0.0.0 (/dev/sg1).
LTFS30207I Vendor ID is QUANTUM.
LTFS30208I Product ID is ULTRIUM-HH7.
LTFS30214I Firmware revision is MA71.
LTFS30215I Drive serial is 90WT021621.
LTFS30285I The reserved buffer size of /dev/sg1 is 1048576.
LTFS30294I Setting up timeout values from RSOC.
LTFS17160I Maximum device block size is 1048576.
LTFS11330I Loading cartridge.
LTFS11332I Load successful.
LTFS17157I Changing the drive setting to write-anywhere mode.
LTFS15049I Checking the medium (mount).
LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 10.
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS14013E Cannot mount the volume from device.
@piste-jp
Copy link
Member

piste-jp commented Oct 7, 2024 •
edited
Loading

What is the iSCSI tape drive you mentioned? Please tell us what it is more specific.

  • Is that provided as a iSCSI tape product from a vendor?
  • Is that a virtual tapa server product (or open source project) ? (like StarWind or mhvtl)
  • Is that configured as an iSCSI tape using real H/W and iSCSI target S/W (by passthrough device) ?

@saltarob
Copy link
Author

saltarob commented Oct 7, 2024 •
edited
Loading

Thank you very much for your quick response.

I am using Linux LIO iSCSI with targetcli on a Proxmox Debian node. The backstore is set up as pscsi (passthrough device). On the AlmaLinux 8.10 VM, I am using Open-iSCSI as the initiator. I hope this information helps. Do you need any additional details about the Quantum LTO-7 HH or the ATTO ExpressSAS H680?

@piste-jp
Copy link
Member

piste-jp commented Oct 7, 2024

This comment is not an official statement and completely based on my experience.

I made a quick experiment on the scsi-target-utils in RHEL5 before and it works. Also I made it on the targetcli in RHEL8 at the same timing, but pscsi cannot handle tape device correctly at that timing.

From the log you provided, sg device cannot report correct transfer length of data. I think this kind of incorrect length report cannot be handled in LTFS. I think you need to dig the iSCSI target or the iSCSI initiator for resolving the issue.

LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 10.
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS14013E Cannot mount the volume from device.

@saltarob
Copy link
Author

saltarob commented Oct 7, 2024

Thank you for your help.

I am trying to set up tgt to verify your findings. As far as I understand, tgt should be equivalent to the scsi-target-utils that you used. However, I am having trouble configuring the tgt target correctly so that the iSCSI initiator and LTFS can recognize and communicate with the tape drive. Since you successfully configured scsi-target-utils, would it be possible for you to assist me in getting the tgt setup working correctly?

Here is my current target configuration for the tape drive:

LUN: 2
Type: passthrough
SCSI ID: IET     00010002
SCSI SN: beaf12
Size: 0 MB
Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: sg
Backing store path: /dev/sg0
Backing store flags:

Issue Details:

  • The /dev/sg0 device appears on the initiator side, but /dev/st0 (the tape device) does not.

  • Running tgtadm does not allow me to add the /dev/st0 device.

  • Here's the output from listing the devices inside the VM:

[root@almalinux-vm saltanat]# ls /dev/sg0 /dev/st0
ls: cannot access '/dev/st0': No such file or directory
/dev/sg0
  • The ltfs -o device_list command does not show any tape devices:
[root@almalinux-vm saltanat]# ltfs -o device_list
7966 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
7966 LTFS14058I LTFS Format Specification version 2.4.0.
7966 LTFS14104I Launched by "ltfs -o device_list".
7966 LTFS14105I This binary is built for Linux (x86_64).
7966 LTFS14106I GCC version is 8.5.0 20210514 (Red Hat 8.5.0-22).
7966 LTFS17087I Kernel version: Linux version 4.18.0-553.22.1.el8_10.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #1 SMP Tue Sep 24 05:16:59 EDT 2024 i386.
7966 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7966 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7966 LTFS17089I Distribution: NAME="AlmaLinux".
7966 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7966 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7966 LTFS17085I Plugin: Loading "sg" tape backend.
Tape Device list:.

@saltarob
Copy link
Author

saltarob commented Oct 7, 2024

Okay, finally I got tgt working. Somehow I didn’t catch that the device changed from sg0 to sg3.

Anyway, I ran the test again with tgt, but it still failed with the same error. Before this, I also tried using the ATTO Xtend SAN iSCSI Initiator for Mac systems, and it failed with the same error too.

The iSCSI connection works perfectly with Bacula and Archiware, so I’m starting to think it’s probably not the iSCSI target or initiator causing the problem. What do you think?

Here’s the output for the ltfs -o devname=/dev/sg3 /ltfsmount command with tgt:

[root@almalinux-vm saltanat]# ltfs -o devname=/dev/sg3 /ltfs
7b76 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
7b76 LTFS14058I LTFS Format Specification version 2.4.0.
7b76 LTFS14104I Launched by "ltfs -o devname=/dev/sg3 /ltfs".
7b76 LTFS14105I This binary is built for Linux (x86_64).
7b76 LTFS14106I GCC version is 8.5.0 20210514 (Red Hat 8.5.0-22).
7b76 LTFS17087I Kernel version: Linux version 4.18.0-553.22.1.el8_10.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #1 SMP Tue Sep 24 05:16:59 EDT 2024 i386.
7b76 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7b76 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7b76 LTFS17089I Distribution: NAME="AlmaLinux".
7b76 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7b76 LTFS17089I Distribution: AlmaLinux release 8.10 (Cerulean Leopard).
7b76 LTFS14063I Sync type is "time", Sync time is 300 sec.
7b76 LTFS17085I Plugin: Loading "sg" tape backend.
7b76 LTFS17085I Plugin: Loading "unified" iosched backend.
7b76 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
7b76 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg3).
7b76 LTFS30250I Opened the SCSI tape device 7.0.0.2 (/dev/sg3).
7b76 LTFS30207I Vendor ID is QUANTUM .
7b76 LTFS30208I Product ID is ULTRIUM-HH7     .
7b76 LTFS30214I Firmware revision is MA71.
7b76 LTFS30215I Drive serial is 90WT021621.
7b76 LTFS30285I The reserved buffer size of /dev/sg3 is 1048576.
7b76 LTFS30294I Setting up timeout values from RSOC.
7b76 LTFS17160I Maximum device block size is 1048576.
7b76 LTFS11330I Loading cartridge.
7b76 LTFS11332I Load successful.
7b76 LTFS17157I Changing the drive setting to write-anywhere mode.
7b76 LTFS11005I Mounting the volume from device.
7b76 LTFS30216W Length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523235).
7b76 LTFS30216W Length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523235).
7b76 LTFS12049E Cannot read: backend call failed (-21716).
7b76 LTFS17039E XML parser: failed to read a block from the medium (-21716).
7b76 LTFS17037E XML parser: failed to read from XML stream.
7b76 LTFS17273E XML parser: Detected an I/O error on reading an index from the tape (-21716).
7b76 LTFS17016E Cannot parse index direct from medium (-21716).
7b76 LTFS11194W Cannot read index: failed to read and parse XML data (-21716).
7b76 LTFS11024E Cannot mount volume: read index failed on the index partition.
7b76 LTFS14013E Cannot mount the volume from device.
7b76 LTFS30252I Logical block protection is disabled.
[root@almalinux-vm saltanat]#

@saltarob
Copy link
Author

saltarob commented Oct 7, 2024

New Insights: I have now compiled LTFS for Debian to run on the Proxmox node with direct hardware access to the tape drive, but I encountered the exact same error. This suggests that the issue has nothing to do with iSCSI. So, what is the problem then?

Here is the output of ltfs -o devname=/dev/sg7 /ltfs on the Proxmox host:

root@pvemacpro:~# ltfs -o devname=/dev/sg7 /ltfs
a808 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
a808 LTFS14058I LTFS Format Specification version 2.4.0.
a808 LTFS14104I Launched by "ltfs -o devname=/dev/sg7 /ltfs".
a808 LTFS14105I This binary is built for Linux (x86_64).
a808 LTFS14106I GCC version is 12.2.0.
a808 LTFS17087I Kernel version: Linux version 6.8.12-2-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) i386.
a808 LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 12 (bookworm)".
a808 LTFS14063I Sync type is "time", Sync time is 300 sec.
a808 LTFS17085I Plugin: Loading "sg" tape backend.
a808 LTFS17085I Plugin: Loading "unified" iosched backend.
a808 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
a808 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg7).
a808 LTFS30250I Opened the SCSI tape device 0.0.0.0 (/dev/sg7).
a808 LTFS30207I Vendor ID is QUANTUM .
a808 LTFS30208I Product ID is ULTRIUM-HH7     .
a808 LTFS30214I Firmware revision is MA71.
a808 LTFS30215I Drive serial is 90WT021621.
a808 LTFS30285I The reserved buffer size of /dev/sg7 is 524288.
a808 LTFS30294I Setting up timeout values from RSOC.
a808 LTFS17160I Maximum device block size is 1048576.
a808 LTFS11330I Loading cartridge.
a808 LTFS11332I Load successful.
a808 LTFS17157I Changing the drive setting to write-anywhere mode.
a808 LTFS11005I Mounting the volume from device.
a808 LTFS30216W Length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523235).
a808 LTFS30216W Length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523235).
a808 LTFS12049E Cannot read: backend call failed (-21716).
a808 LTFS17039E XML parser: failed to read a block from the medium (-21716).
a808 LTFS17037E XML parser: failed to read from XML stream.
a808 LTFS17273E XML parser: Detected an I/O error on reading an index from the tape (-21716).
a808 LTFS17016E Cannot parse index direct from medium (-21716).
a808 LTFS11194W Cannot read index: failed to read and parse XML data (-21716).
a808 LTFS11024E Cannot mount volume: read index failed on the index partition.
a808 LTFS14013E Cannot mount the volume from device.
a808 LTFS30252I Logical block protection is disabled.

@saltarob
Copy link
Author

saltarob commented Oct 7, 2024

Is the note for buggy HBAs related to my problem?

https://github.com/LinearTapeFileSystem/ltfs/wiki/HBA-info#note-for-buggy-hbas

@piste-jp
Copy link
Member

piste-jp commented Oct 7, 2024

From the lines like below, I think --enable-buggy-ifs highly likely works.

a808 LTFS30216W Length mismatch is detected. (Act = 524288, resid = 0, resid_sense = 523235).

See also, #369 (comment).

@saltarob
Copy link
Author

saltarob commented Oct 8, 2024

Great! The --enable-buggy-ifs option is working with both direct hardware connections and through iSCSI using targetcli. Thank you very much.

I have been using the ATTO ExpressSAS H680 for years on a macOS system with LTFS and YoYotta, and I have never encountered such a problem. According to your description, the error seems to come from the HBA not being able to handle the transfer length of the SCSI data. Now, I’m wondering if the problem is really rooted in the hardware or possibly in the driver. On macOS, I use the driver provided by ATTO for the HBA. Unfortunately, ATTO doesn’t offer drivers for Debian or any newer Linux systems, so I’m relying on the pm80xx driver. If it is a driver issue, perhaps it should be reported to the developers of pm80xx?

@piste-jp
Copy link
Member

piste-jp commented Oct 8, 2024 •
edited
Loading

If it is a driver issue, perhaps it should be reported to the developers of pm80xx?

I'm not sure it should be or not. But first candidate is reporting this to ATTO, second one might be pm80xx as you said.

@saltarob
Copy link
Author

saltarob commented Oct 9, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@saltarob @piste-jp