Replies: 16 comments 5 replies
-
|
Can you edit the description to make more clear what the issue is? |
Beta Was this translation helpful? Give feedback.
-
|
when i add secondary storage using NFS to zone, the KVM host gets stuck in alert state
now i can't deploy VM, register template or do anything else. The problem was only resolved when I removed the secondary storage from the zone. I have another zone, it uses ceph for primary storage and doesn't have the above problem. |
Beta Was this translation helpful? Give feedback.
-
|
So when you add a secondary storage to an already enabled zone, @leduyquy ? |
Beta Was this translation helpful? Give feedback.
-
|
Can the host reach the NFS? Can you share the agent log for the host?
here is the log from the management server i found. hope it helps
|
Beta Was this translation helpful? Give feedback.
-
|
@leduyquy , it seems |
Beta Was this translation helpful? Give feedback.
-
|
I don't use KVM migration so I didn't install it, Other hosts also do not need to install this package and it still works fine. After I installed it, the log also no longer has the alert, but the host is still stuck in the alert. Log on agent:
|
Beta Was this translation helpful? Give feedback.
-
|
@leduyquy , on the host (and compare it to the output on the other hosts)? |
Beta Was this translation helpful? Give feedback.
-
|
I don't think it's related to the problem I'm having. I see it will get GHz info from CPU Model name, however for gold series CPU it has removed it from CPU name. In host problem:
In another host:
I think there is a problem with secondary storage, because when I remove it from the zone, the host changes to state: UP |
Beta Was this translation helpful? Give feedback.
-
|
sounds reasonable, but as a followup, do the rest of the hosts all look the same? it must be the combination host/secondary storage as far as I understand, if all other hosts function normally. |
Beta Was this translation helpful? Give feedback.
-
|
btw, can you also check nfs versions used on the server and the clients? |
Beta Was this translation helpful? Give feedback.
-
|
All kvm hosts use NFSv4 and i have never had any problems with it before I tried another test case. I added a primary storage using NFS to the above zone and it worked fine without any problems. when i add nfs as secondary storage i notice it mounts to manager and kvm host, however it is removed from manager and host after the process of syncing templates to this secondary storage is complete. Then the state of kvm host agent will change to alert |
Beta Was this translation helpful? Give feedback.
-
|
Update: I checked further and found that it is only mounted nfs secondary storage at Secondary Storage VMs |
Beta Was this translation helpful? Give feedback.
-
this seems alright. it should not be the reason the host goes to alert state. |
Beta Was this translation helpful? Give feedback.
-
|
maybe we need to check what triggered the kvm agent investigation @leduyquy |
Beta Was this translation helpful? Give feedback.
-
|
I have setup a new zone and this is the log related to secondary storage when I create the zone. if you need any more logs please give me more info about it, there are quite a few logs and it's confusing me Note: I have to change the system template because my system is isolated from the internet so I need to download it to a local repository and then register the system template from there to avoid some errors when it tries to download it from the internet.
|
Beta Was this translation helpful? Give feedback.
-
|
this log
indicates that either the host or ssvm is down, OR cannot reach the management server over the management network. There is a ssvm-check.sh on the ssvm to get some diagnostics. as per
the ssvm is probably not started. The secondary storage needs seeding and as you state this environment is not connected to the internet you need to do it by hand. (see https://docs.cloudstack.apache.org/en/4.20.0.0/installguide/management-server/index.html#prepare-the-system-vm-template) but as you state you use 4.20 , this should not be a problem. Can you check the mount ( |
Beta Was this translation helpful? Give feedback.
-
|
@leduyquy |
Beta Was this translation helpful? Give feedback.
-
This is output when check with ssvm-check.sh
because i set firewall for dns traffic only, so ICMP traffic will be dropped
It is absolutely correct, and I named it like that to tell the difference between test cases. when i check with ssvm it also has mountpoint
both SSVM and CPVM are running and state UP, but when i try to destroy it it won't redeploy like normal |
Beta Was this translation helpful? Give feedback.
-
So you can do any queries from the ssvm?
how about the rights on the sec-stor? and
Do you see any further logs in the management server about trying to start the SVMs again? |
Beta Was this translation helpful? Give feedback.
-
I tried queries some dns requests with local domain and it was fine. However it is deployed in an environment that is not connected to the internet so it will not be able to query any public domains outside the internet.
with the path above it has been removed from the management server so i cant test it. But i check with the mountpoint path on ssvm it seems everything is fine.
I just tried restarting the management service. The secondary is mounted to the management server. Systemvm redeployed after I destroyed it. I can still register templates and create instances normally even though there is no mountpoint of the secondary storage on the kvm host. KVM host agent state: UP. I feel quite confused about it. Maybe I will try to reproduce the problem above. |
Beta Was this translation helpful? Give feedback.
-
|
cloudstack-management service is running with user |
Beta Was this translation helpful? Give feedback.
-
problem
I have plan deploy 1 zone use local disk and GPU, Because i don't need HA, i don't deploy any shared storage to primary storage. I using only local disk for primary storage.
When i setup Secondary storage use NFS for store template and ISO, KVM host in Zone state stuck on alert, To make sure the NFS configuration works fine, I mounted it manually on the kvm host and managerment and it worked fine
when i created the zone i saw it still created 2 system vm for the zone and it worked fine, things only got worse when i started creating VMs and digging deeper into them
versions
Managerment Version: 4.20.0
Agent Version: 4.20.0
KVM host OS: 22.04
The steps to reproduce the bug
...
What to do about it?
Add Secondary storage and anything working fine
Beta Was this translation helpful? Give feedback.
All reactions