Coverband.configuration.store.coverage returns empty hash

[harshanails]

Describe the bug
We use Coverband to map covered file paths in Gitlab's E2E specs.

In this context, we have an internal api which calls Coverband.configuration.store.coverage but it keeps returning {} from the api.

But, from the rails console, the same call Coverband.configuration.store.coverage returns a non-empty coverage hash.

In rails logs I do see the following:

Coverband: background reporting coverage (runtime). Sleeping 3s
Coverband: background reporting coverage (runtime). Sleeping 3s
Coverband: background reporting coverage (runtime). Sleeping 24s
Coverband: background reporting coverage (runtime). Sleeping 24s

So that means, coverband is reporting to redis but the api is just unable to fetch the coverage 🤔

To Reproduce
It is perhaps specific to Gitlab environment, but would like some help in troubleshooting further.

Expected behavior
Coverband.configuration.store.coverage should return a non-empty hash when some coverage was recorded.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[danmayer]

folks use coverband on gitlab and there were a few PRs related to improvements or figuring it out in the past... I would need more specifics, but it is likely that the redis used when reporting is different when trying to retrieve the coverage.

[danmayer]

Do the end to end tests run in the DEV env while the report runs in TEST env? Do you have anything that tries to wipe and clear redis between tests or after test suites or after test files?

[harshanails]

Do the end to end tests run in the DEV env while the report runs in TEST env? Do you have anything that tries to wipe and clear redis between tests or after test suites or after test files?

Yes, but this is not relevant to this issue. Even without running any tests, the coverage returned is an empty hash.

[harshanails]

folks use coverband on gitlab and there were a few PRs related to improvements or figuring it out in the past... I would need more specifics, but it is likely that the redis used when reporting is different when trying to retrieve the coverage.

In order to reproduce the issue you could spin up a gitlab docker instance with

docker run -d --env GITLAB_OMNIBUS_CONFIG='gitlab_rails["initial_root_password"] = "password"'  --name gitlab.local --hostname gitlab-ee-5e359caf.test -p 80 --volume /Users/harshamuralidhar/quality/work/log:/var/log/gitlab:Z --shm-size 256m gitlab/gitlab-ee:nightly

And then set COVERBAND_ENABLED=true in /etc/gitlab/gitlab.rb and restart rails application.

[harshanails]

@danmayer I would like to troubleshoot if the coverage is being reported to redis store at all. How can I check this with redis-cli? Is there a way to check this if I have access to the redis?

[harshanails]

I see in redis logs of the application, this error but I am not sure if it is related to coverband

redis: 2024/10/24 04:06:14 pubsub.go:168: redis: discarding bad PubSub connection: EOF
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2024-10-24T04:06:14Z"}

[harshanails]

@danmayer could you point me to the correct configuration to use file_store instead of redis store? Because I have configured coverband to use Gitlab’s own redis server, it may be causing some issues.

What are the limitations to using file store? Note that we are not using “coverband” to check production code coverage - although we are using it to map code paths to E2E tests on a production application

[danmayer]

sorry that I haven't gotten back to this. I can try to look into it more over the weekend and see if I can figure out that issue with gitlab.

• If you can setup another redis server you could just try using a redis client connection to that... if there are worries about the shared resources
• the filestore will work for a single computer, but if you have more than one it will have issues
• have you seen or tried the memcached store?

[harshanails]

If you can setup another redis server you could just try using a redis client connection to that... if there are worries about the shared resources

No, this is not feasible in our use case. To be clear, coverband works. But for our use case, we need it to be part of the gitlab application and dependency on an external redis server will not pass through reviews.

the filestore will work for a single computer, but if you have more than one it will have issues

Our use case is a single docker image in fact. So this shouldn't be a problem. But I am facing this error:

STDERR: #<Thread:0x00007efcd1164750@Coverband Background Reporter /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/integrations/background.rb:32 run> terminated with exception (report_on_exception is true):
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/adapters/file_store.rb:74:in `raw_store': FileStore doesn't support raw_store (NotImplementedError)
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/collectors/abstract_tracker.rb:141:in `redis_store'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/collectors/abstract_tracker.rb:148:in `tracker_time_key_exists?'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/collectors/abstract_tracker.rb:97:in `save_report'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/integrations/background.rb:43:in `block (4 levels) in start'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/integrations/background.rb:43:in `each'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/integrations/background.rb:43:in `block (3 levels) in start'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/integrations/background.rb:35:in `loop'
	from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/integrations/background.rb:35:in `block (2 levels) in start'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/coverband-6.1.3/lib/coverband/adapters/file_store.rb:74:in `raw_store': FileStore doesn't support raw_store (NotImplementedError)

Here is the MR where I am trying to do that: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/170375/diffs

have you seen or tried the memcached store?

No, what is that?

[danmayer]

Ok so the error you are having with filestore is when trying to use the view tracker and i18n tracker, these build off the raw redis store... If you only need code coverage, you can likely use the filestore along with this configuration options:

config.track_views = false

If you want to track views or the other options like translations or routes, you will need redis (or I need to support them with other adapters).

If you want to better debug redis, can you use a Rails command line that is easier than the redis-cli to ensure you have the right namespace and configured redis.

> Coverband.configuration.store
 =>
#<Coverband::Adapters::RedisStore:0x00007f68558f24f0
 @base_key=nil,
 @format_version="coverband_3_2",
 @keys={:runtime=>"coverband_3_2.runtime", :eager_loading=>"coverband_3_2.eager_loading"},
 @redis=#<Redis client v5.1.0 for redis://localhost:6379/0>,
 @redis_namespace=nil,
 @ttl=2592000,
 @type=:runtime>
> Coverband.configuration.store.coverage
=>  =>
{"./config/routes.rb"=>
  {"first_updated_at"=>1730174741,
   "last_updated_at"=>1730175047,
   "file_hash"=>"68c81b8a494b4836eb597b0a7a411d21",
   "data"=>[3, 3, 3, nil, nil, nil, 3, nil, nil, 3, nil, nil, nil, nil, nil, nil, nil, nil, 3, nil]},
 "./app/controllers/books_controller.rb"=>
  {"first_updated_at"=>1730174741,
   "last_updated_at"=>1730174752,
   "file_hash"=>"d1b4be779b793d47ce5f07e576ad53f3",
   "data"=>
    [0,
...

If you really need to use the CLI, where to look depends on a few options and which redis adapter you are using the one above or the HashRedisStore which uses LUA.

for the normal adapter you can look somewhere like below which will spit out a giant JSON blob

 redis-cli
127.0.0.1:6379> get coverband_3_2.runtime
"{\"./config/routes.rb\":{\"first_updated_at\":1730174741,\"last_upd.....

If your redis is local to the docker you need to make sure your context either your console or whatever is the same docker and same redis or you won't find data... If you can tail your redis commands you could also look for something in tests clearing the redis as that seems like a possibility.

[harshanails]

If you only need code coverage, you can likely use the filestore along with this configuration options:

After evaluating file store option, we think it doesn't serve our use case. The redis store is working in two of our build environments but just not in omnibus docker environment.

Context:
We use an internal api to GET the coverage from coverband. For example you can see in this job output where the api is getting called, it returns an empty array in this case.

In redis-cli I was able to confirm that the data is indeed being written to coverband_3_2.runtime but I am not sure why it isn't being fetched with ::Coverband.configuration.store.coverage.

In order run the application as omnibus docker locally, the command is mentioned in this comment

Thank you for looking into this.

[harshanails]

@danmayer What are the scenarios in which ::Coverband.configuration.store.coverage returns an empty hash? - given that the coverage is being reported correctly to coverband_3_2.runtime .

We changed this method call to include skip_hash_check: true to fix it in one of our deployed environments.