Skip to content
This repository has been archived by the owner on Dec 13, 2022. It is now read-only.

Resource Usage in dashboard saying something went wrong with version v4.2.2 #940

Open
thenucleophile opened this issue Apr 25, 2014 · 10 comments
Open

Resource Usage in dashboard saying something went wrong with version v4.2.2 #940

thenucleophile opened this issue Apr 25, 2014 · 10 comments
Milestone
v4.2.3

Comments

@thenucleophile
Copy link

Resource Usage in dashboard saying something went wrong with version v4.2.2
while ceilometer resource-list is giving output as below

root@controller:~# ceilometer resource-list
+----------------------------------------------+--------+----------------------------------+----------------------------------+
| Resource ID | Source | User ID | Project ID |
+----------------------------------------------+--------+----------------------------------+----------------------------------+
| 50a9a09e-a71a-477e-816a-3ac8a8a1b055 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| 6c794811-73fa-4f4c-bef4-ce5fea20d9fc | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 |
| 8813ff3d-6b65-45c5-b0ca-f104bab41f7c | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 |
| 8f81eefb-5f28-4b5c-b283-58911710adcd | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| 99344452-5fee-4607-acb7-74b257bd948a | | None | 5b3463337426443fbc85633051c2e972 |
| 9d16cb95-fd6d-4132-8a5c-ec06ebbb7319 | | None | 5b3463337426443fbc85633051c2e972 |
| aff4e933-8e83-4aa0-83c0-c9a559dacdaa | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| b1206217-cc37-4780-a882-6013b885cc79 | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| b912ab7f-380a-49dd-abe6-347635660429 | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 |
| cbc0c113-45e2-427b-8ec4-8ddf3a3c6bf5 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| cbf18e9f-33ed-439b-89e7-a8f106fdc537 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| cc3f8efa-ed18-49b1-b696-2fa3a58115a8 | | None | 5b3463337426443fbc85633051c2e972 |
| d33f8089-b751-456f-9054-98de74560feb | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| d85e264d-8bef-4679-b17f-65575192f558 | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| d919a729-90f3-46d7-994d-86371c47b37b | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| e9e0e8ba-4420-4b05-9759-1c68c626356c | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| f2f30735-196f-4db8-90d7-9b9eb6eca732 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| nova-instance-instance-00000001-fa163eb56aa0 | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 |
| nova-instance-instance-00000003-fa163e9a105d | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| nova-instance-instance-00000005-fa163e7b328f | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 |
| nova-instance-instance-0000000b-fa163efdd281 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
| nova-instance-instance-0000001b-fa163e2ed3ee | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf |
+----------------------------------------------+--------+----------------------------------+----------------------------------+

root@controller:/var/log/ceilometer# cat ceilometer-api.log
2014-04-25 11:02:48.869 7826 ERROR wsme.api [-] Server-side error: "metaquery not implemented". Detail:
Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/wsmeext/pecan.py", line 72, in callfunction
result = f(self, _args, *_kwargs)

File "/usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py", line 968, in get_all
for r in pecan.request.storage_conn.get_resources(**kwargs)]

File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 310, in get_resources
raise NotImplementedError(_('metaquery not implemented'))

NotImplementedError: metaquery not implemented

Output of cat ceilometer-agent-compute.log

2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return connection.drain_events(**kwargs)
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common chanmap, None, timeout=timeout,
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.method_reader.read_method()
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common raise m
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common error: [Errno 110] Connection timed out
2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common
2014-04-25 11:29:13.813 29155 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.20.0.175:5672
2014-04-25 11:29:13.833 29155 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.20.0.175:5672

Please help!
@cloudnull
Copy link
Contributor

Couple of questions:

  • what os are you running?
  • have the packages been upgraded outside of the cookbooks and the installation?
  • when was the installation done?
  • Is this an HA setup?
  • what is the version of the packages you have installed (dpkg -l | grep ceilometer || rpm -qa | grep ceilometer)

Get back to us with all of the information you can about your environment and we'll see what needs to be done to get it back online.

@thenucleophile
Copy link
Author

Ubuntu 12.04
upgraded packages after cookbook installation initially it was 4.2.0
installation was done 1 month back
It is not HA setup. One of each chef server, controller node (chef-client) & compute node (chef-client)

Controller Node:
root@controller:# dpkg -l | grep ceilometer || rpm -qa | grep ceilometer
ii ceilometer-agent-central 2013.2.2-0ubuntu1cloud0 ceilometer central agent
rc ceilometer-agent-compute 2013.2.2-0ubuntu1cloud0 ceilometer compute agent
ii ceilometer-api 2013.2.2-0ubuntu1cloud0 ceilometer api service
ii ceilometer-collector 2013.2.2-0ubuntu1cloud0 ceilometer collector service
ii ceilometer-common 2013.2.2-0ubuntu1cloud0 ceilometer common files
ii python-ceilometer 2013.2.2-0ubuntu1cloud0 ceilometer python libraries
ii python-ceilometerclient 1.0.5-0ubuntu1cloud0 Client library for Openstack ceilometer server.

Compute Node:
root@compute:# dpkg -l | grep ceilometer || rpm -qa | grep ceilometer
ii ceilometer-agent-compute 2013.2.2-0ubuntu1cloud0 ceilometer compute agent
ii ceilometer-common 2013.2.2-0ubuntu1cloud0 ceilometer common files
ii python-ceilometer 2013.2.2-0ubuntu1cloud0 ceilometer python libraries
ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

It is showing something went wrong in dashboard in resource usage tab

root@controller:/var/log/ceilometer# cat ceilometer-api.log

2014-04-28 11:29:39.235 1207 ERROR wsme.api [-] Server-side error: "metaquery not implemented". Detail:
Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/wsmeext/pecan.py", line 72, in callfunction
result = f(self, _args, *_kwargs)

File "/usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py", line 968, in get_all
for r in pecan.request.storage_conn.get_resources(**kwargs)]

File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 310, in get_resources
raise NotImplementedError(_('metaquery not implemented'))

NotImplementedError: metaquery not implemented

As per below link
#799

this problem should not come as cookbook version is upgraded to 4.2.2

Please help! Thanks.

@thenucleophile
Copy link
Author

i installed ceilometer-agent-compute on controller by mistake..it is already there on compute node...it can be removed as well..but resource usage tab doesn't appear..

Also all services are running fine.

root@controller:/etc/nova# nova-manage service list
Binary Host Zone Status State Updated_At
nova-conductor controller internal enabled :-) 2014-04-28 12:12:57
nova-cert controller internal enabled :-) 2014-04-28 12:12:58
nova-consoleauth controller internal enabled :-) 2014-04-28 12:12:58
nova-scheduler controller internal enabled :-) 2014-04-28 12:12:58
nova-network compute internal enabled :-) 2014-04-28 12:13:04
nova-compute compute nova enabled :-) 2014-04-28 12:12:08

@thenucleophile
Copy link
Author

Do not why but nova-network is going down after sometime after rebooting compute node.

root@controller:/etc# nova-manage service list
Binary Host Zone Status State Updated_At
nova-conductor controller internal enabled :-) 2014-04-29 07:02:52
nova-cert controller internal enabled :-) 2014-04-29 07:03:01
nova-consoleauth controller internal enabled :-) 2014-04-29 07:03:00
nova-scheduler controller internal enabled :-) 2014-04-29 07:02:58
nova-network compute internal enabled XXX 2014-04-29 06:25:37
nova-compute compute nova enabled :-) 2014-04-29 07:02:52

@breu
Copy link
Contributor

breu commented Apr 29, 2014

Ubuntu and RedHat packages were updated that caused our hand patch of ceilometer to fail.

RedHat
openstack-ceilometer-collector-2013.2.3-1.el6.noarch
python-ceilometerclient-1.0.8-1.el6.noarch
python-ceilometer-2013.2.3-1.el6.noarch
openstack-ceilometer-api-2013.2.3-1.el6.noarch
openstack-ceilometer-common-2013.2.3-1.el6.noarch
openstack-ceilometer-central-2013.2.3-1.el6.noarch

Ubuntu
ii ceilometer-agent-central 2013.2.2-0ubuntu1~cloud0 ceilometer central agent
rc ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent
ii ceilometer-api 2013.2.2-0ubuntu1~cloud0 ceilometer api service
ii ceilometer-collector 2013.2.2-0ubuntu1~cloud0 ceilometer collector service
ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files
ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries
ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

We will need to patch the patcher again..

@breu
Copy link
Contributor

breu commented Apr 29, 2014

@thenucleophile Your nova-network problem is something separate from ceilometer. Check the nova-network log file but don't lump this into the same github issue.

@thenucleophile
Copy link
Author

@rackerjoe Thanks!! Please update once the patcher is patched..

@sstent
Copy link

sstent commented May 20, 2014

I'm seeing this on Rhel 6.5 as well

@claco claco added this to the v4.2.3 milestone May 23, 2014
@cloudnull cloudnull mentioned this issue May 23, 2014
Closed
@claco claco mentioned this issue May 26, 2014
Merged
@runningman84
Copy link

Any news here? This problem still exists in Ubuntu 12.04 LTS

@claco
Copy link
Contributor

claco commented Jun 20, 2014

@runningman84 Yes! There's a patch merged into master and the v4.2.3rc branches of ceilometer to fix this:

rcbops-cookbooks/ceilometer#48
https://github.com/rcbops-cookbooks/ceilometer/tree/v4.2.3rc

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v4.2.3
Development

No branches or pull requests
6 participants
@claco @breu @sstent @runningman84 @cloudnull @thenucleophile