Danube giving a new node an old hostname

asked 2017-10-11 10:31:50 -0700

jokken gravatar image

updated 2017-10-11 10:35:02 -0700

hi all

I am using Danube Fuel 10. I have a known good environment, I've used fuel to deploy the controllers, etc and 10 compute nodes so far. I have instances running on those nodes..

when attempting to deploy a 11th compute node I am running into issues. Way back, before the 10 compute nodes, there was a previous attempt to deploy this node, this attempt was unsuccessful because of a bad hard drive. Now that this is fixed I am deploying again.

Back then Fuel refered to it as hostname node-20. Now fuel is using hostname node-31. Everything I see on the surface, including the fuel2 CLI, shows this node as node-31. Yet, after Fuel provisions the Ubuntu OS and the host reboots cloud-init (i believe) is setting the old node-20 hostname, and using many settings from node-20 (ie: RSA SSH keys).

I believe this is causing Fuel not to be able to communicate with the node and the deployment times out with #<runtimeerror: could="" not="" find="" any="" hosts="" in="" discover="" data="" provided="">

[fuel2 node show 31
| id | 31 |
| name | Untitled (68:58) |
| status | ready |
| os_platform | ubuntu |
| roles | [u'compute'] |
| kernel_params | None |
| pending_roles | [] |
| hostname | node-31 |
| fqdn | node-31.ourdomain.com |
| platform_name | ProLiant BL460c Gen9 |

on this broken node-31 I find these files/values in /var/lib/cloud

root@node-20:/var/lib/cloud# grep -R node-20 *

Binary file instance/obj.pkl matches
instance/user-data.txt:hostname: node-20.ourdomain.com
instance/user-data.txt:fqdn: node-20.ourdomain.com
instance/user-data.txt.i:hostname: node-20.ourdomain.com
instance/user-data.txt.i:fqdn: node-20.ourdomain.com
instance/cloud-config.txt:fqdn: node-20.ourdomain.com
instance/cloud-config.txt:hostname: node-20.ourdomain.com
Binary file instances/nocloud/obj.pkl matches
instances/nocloud/user-data.txt:hostname: node-20.ourdomain.com
instances/nocloud/user-data.txt:fqdn: node-20.ourdomain.com
instances/nocloud/user-data.txt.i:hostname: node-20.ourdomain.com
instances/nocloud/user-data.txt.i:fqdn: node-20.ourdomain.com
instances/nocloud/cloud-config.txt:fqdn: node-20.ourdomain.com
instances/nocloud/cloud-config.txt:hostname: node-20.ourdomain.com

please let me know what in Fuel 10 is setting these values? is it cloud-init? And how can I intervene and avoid it being set incorrectly?

If I try to delete node-20 I get an error showing it doesn't exist from fuel/2's perspective:

fuel node --node-id 20 --delete-from-db --force
404 Client Error: Not Found for url: http://10.20.20.1:8000/api/v1/nodes/?ids=20 (NodeCollection not found)
fuel2 node undiscover -n 20 -f
404 Client Error: Not Found for url: http://10.20.20.1:8000/api/v1/nodes/20/ (Node not found)

I guess this has something to do with this file or one similar:

/usr/share/fuel-agent/cloud-init-templates/cloud_config_ubuntu.jinja2
in that file I see
hostname: {{ common.hostname }}
fqdn: {{ common.fqdn }}

I assume the node's UUID or MAC address is already in some Fuel database associated with node-20, and these files are being populated incorrectly because of that.

I've tried many different things to get passed this ... (more)

edit retag flag offensive close merge delete