TL;DR Shutdown all VMs than run modprobe -r kvm_intel and modprobe kvm_intel nested=1 in unraid terminal. After that add <feature policy='require' name='vmx'/> to the CPU Section of target VM.

VM

Make sure that all VM are stopped. Wait a little bit after stopping.

Open the Unraid Terminal

Run the following via a terminal in unraid (SpaceInvader’s thank you!)

#!/bin/bash

modprobe -r kvm_amd
modprobe kvm_amd nested=1
echo "Nested vms are enabled for AMD cpus"

sleep 10
exit

Open VM Settings -> Toggle XML View

Add feature to to the CPU Section

<feature policy='require' name='vmx'/>

Tips

Any way to reverse these commands?

modprobe -r kvm_amd
modprobe kvm_amd nested=0