Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SequenceSearchDef: report file search progress on debug log #16

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

xmkg
Copy link
Contributor

@xmkg xmkg commented May 9, 2024

shorten the search file name to just parent_folder/filename and also print the search def tags before the search, so we can tell which search is currently being run.

Example log output:

2024-05-09 14:03:14,667 268633 DEBUG searchkit [-] starting search of /workspace/case-data/test/sosreport-juju-4b6be0-3-lxd-13-00359298-2023-05-02-tiaplar/var/log/neutron/neutron-server.log.1 (offset=0, pos=0) for tags [{'openstack.neutron.ovndb_leader_bouncing.checks.has_nbdb_log.check.search', 'openstack.neutron.ovn_stale_db.checks.has_1829109_neutron_log.check.search', 'openstack.neutron.ovndb_leader_bouncing.checks.has_sbdb_log.check.search', 'openstack.neutron.bugs.lp1993628.checks.logs.check.search', 'openstack.neutron.bugs.lp1980211.checks.has_ovsdb_reconnection_errors.check.search', 'openstack.neutron.bugs.lp1948466.checks.has_1948466.check.search', 'openstack.neutron.bugs.lp1960319.checks.has_1960319_log.check.search'}]
2024-05-09 14:03:14,667 268632 DEBUG searchkit [-] starting search of /workspace/case-data/test/sosreport-juju-4b6be0-3-lxd-13-00359298-2023-05-02-tiaplar/var/log/neutron/neutron-server.log (offset=0, pos=0) for tags [{'openstack.neutron.ovndb_leader_bouncing.checks.has_nbdb_log.check.search', 'openstack.neutron.ovn_stale_db.checks.has_1829109_neutron_log.check.search', 'openstack.neutron.ovndb_leader_bouncing.checks.has_sbdb_log.check.search', 'openstack.neutron.bugs.lp1993628.checks.logs.check.search', 'openstack.neutron.bugs.lp1980211.checks.has_ovsdb_reconnection_errors.check.search', 'openstack.neutron.bugs.lp1948466.checks.has_1948466.check.search', 'openstack.neutron.bugs.lp1960319.checks.has_1960319_log.check.search'}]
/*...*/
2024-05-09 14:03:18,280 268632 DEBUG searchkit [-] 100000 lines searched in neutron/neutron-server.log (0.304459%)

# NOTE: line numbers start at 1 hence offset + 1
for ln, line in enumerate(fd, start=offset + 1):
# This could be helpful to show progress for large files
if ln % 100000 == 0:
log.debug("%s lines searched in %s", ln, fd.name)
log.debug("%s lines searched in %s (%g%%)",
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we round the pcent to 3 dp e.g. {0:.3f}

Copy link
Contributor Author

@xmkg xmkg May 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done:

2024-05-09 15:21:00,860 544680 DEBUG searchkit [-] 200000 lines searched in openstackrgxrh5t2/tmpry787c7r (55.455%)
2024-05-09 15:21:00,947 544681 DEBUG searchkit [-] 300000 lines searched in openstackrgxrh5t2/tmpcqx172jh (82.786%)
2024-05-09 15:21:00,954 544680 DEBUG searchkit [-] 300000 lines searched in openstackrgxrh5t2/tmpry787c7r (82.786%)

log.debug("starting search of %s (offset=%s, pos=%s)", fd.name, offset,
fd.tell())
log.debug("starting search of %s (offset=%s, pos=%s) for tags [%s]",
fd.name, offset, fd.tell(), search_tags)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is nice to see the tags but maybe we need to format it differently to make it readable since it will be a very very long line otherwise.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done:

2024-05-09 15:20:57,660 544681 DEBUG searchkit [-] starting search of /workspace/case-data/test/sosreport-juju-4b6be0-3-lxd-13-00359298-2023-05-02-tiaplar/var/log/neutron/neutron-server.log.1 (offset=0, pos=0) for tags [[
  "openstack.neutron.ovn_stale_db.checks.has_1829109_neutron_log.check.search",
  "openstack.neutron.bugs.lp1948466.checks.has_1948466.check.search",
  "openstack.neutron.bugs.lp1980211.checks.has_ovsdb_reconnection_errors.check.search",
  "openstack.neutron.bugs.lp1960319.checks.has_1960319_log.check.search",
  "openstack.neutron.bugs.lp1993628.checks.logs.check.search",
  "openstack.neutron.ovndb_leader_bouncing.checks.has_nbdb_log.check.search",
  "openstack.neutron.ovndb_leader_bouncing.checks.has_sbdb_log.check.search"
]]

@xmkg xmkg force-pushed the enhancement/debug-file-search-progress branch 3 times, most recently from 0337068 to a053a45 Compare May 9, 2024 12:20
@xmkg xmkg requested a review from dosaboy May 9, 2024 12:22
@xmkg xmkg force-pushed the enhancement/debug-file-search-progress branch 4 times, most recently from 7a6c769 to 4e8542c Compare May 13, 2024 11:57
shorten the search file name to just parent_folder/filename and
also print the search def tags before the search, so we can tell
which search is currently being run.

Signed-off-by: Mustafa Kemal Gilor <[email protected]>
@xmkg xmkg force-pushed the enhancement/debug-file-search-progress branch from 4e8542c to 31f8c87 Compare May 24, 2024 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants