new system project

[driz 2626]

I'm building a new system for my girls, i needed more server power and took back their other system. 

 

http://a.co/fiePrHf

 

it comes out to 343.98 with rebates/discounts.  I was forced to use AMD cards because nvidia only supports pci passthru on grid and quadra series cards. basically, i'll run linux with kvm on there and 2 win10 virtual machines. each machine will have a dedicated USB controller (2 ports) and a dedicated graphics card. the systems will each have 6GB ram and 1 cpu core (plus the hyperthreads). 

 

the system will be used for MOSTLY playing minecraft on max quality (which isn't saying much) and web based games like nickjr.com and shit like that.

 

I have a power supply sitting in my storage box, so that is why one isn't included.

Storage will be network attached ssd's

[Swed 2651]

1 hour ago, driz said:

I'm building a new system for my girls, i needed more server power and took back their other system. 

 

http://a.co/fiePrHf

 

it comes out to 343.98 with rebates/discounts.  I was forced to use AMD cards because nvidia only supports pci passthru on grid and quadra series cards. basically, i'll run linux with kvm on there and 2 win10 virtual machines. each machine will have a dedicated USB controller (2 ports) and a dedicated graphics card. the systems will each have 6GB ram and 1 cpu core (plus the hyperthreads). 

 

the system will be used for MOSTLY playing minecraft on max quality (which isn't saying much) and web based games like nickjr.com and shit like that.

 

I have a power supply sitting in my storage box, so that is why one isn't included.

What about pixelmon? I allocate 12GB Ram for that shit. @Rayne

[FreeLancer 44]

All the parts seem to be fine, and can handle those games. Go for it!

You might need to add a hard drive if you dont already have one.

[driz 2626]

3 minutes ago, FreeLancer said:

All the parts seem to be fine, and can handle those games. Go for it!

You might need to add a hard drive if you dont already have one.

I have a bank of SSDs in a SAN that will be attached to the guest machines. no physical disks will be in this system.

10 minutes ago, Swed said:

What about pixelmon? I allocate 12GB Ram for that shit. @Rayne

is pixelmon a minecraft thing?

[Swed 2651]

3 minutes ago, driz said:

I have a bank of SSDs in a SAN that will be attached to the guest machines. no physical disks will be in this system.

is pixelmon a minecraft thing?

A huge minecraft mod. Yep.

[centran 4457]

it's no microwave but I guess it'll do

[spiderpigrider 312]

Can we get a picture  Of the final product?

[Rayne 4478]

10 hours ago, driz said:

is pixelmon a minecraft thing?

 

That eats your memory. Good fun though.

[cookie eater 965]

looks good driz, 

nice cheap pc

[driz 2626]

14 hours ago, spiderpigrider said:

Can we get a picture  Of the final product?

it'll look like a black box with some cables coming out of it... but sure

 

[cookie eater 965]

Hey don't cable manage anything and throw some dust in it. Improves performance. The dust acts like a lubricant and the cable management doesn't do anything so it's a waste of time.

Sent from my Z716BL using Tapatalk

[centran 4457]

17 hours ago, cookie eater said:

Hey don't cable manage anything and throw some dust in it. Improves performance. The dust acts like a lubricant and the cable management doesn't do anything so it's a waste of time.

Sent from my Z716BL using Tapatalk

 

If you put dust in to help performance you also have to remember to punch some holes in the case so that the dust will move around and those lubricate. The industry term for those is speed holes.

[driz 2626]

ok, i know it's been a while, but i actually finished this project. I've got near bare metal performance on both windows guests. total cost was $564.91 (this include rebates of 30$ and an additional 8G of ram that i ordered. This also includes a new office chair and rubber casters)

 

here is the full break down of components and how things are configured.

Motherboard: ASRock Micro ATX H170M PRO4
CPU: Intel 3.7GHz Core i3-6100
RAM: Patriot Extreme Performance 8G PC4-25600 (3200MHz) Viper Elite Series (i just ordered a 2nd kit to give 16G total)
Video: 2x MSI AMD Radeon R7 240 2GB DDR3 Low Profile PCIE
Audio: 2x Sabrent USB External Stero Sound Adapter (C-Media)
Storage: 2x Adata SU800 128G 3D-NAND 2.5" SATA3 SSD
Host Storage: 1x NVMe 250G Samsung 960 EVO

I'm also ordering 2 usb 3 extension cabled and hubs so the girls can plug their shit on on the desk rather than having a mess of cables going to the host system.

Each system has a keyboard, mouse, video card, and usb audio device passed through directly to the guest windows 10 machine.

 

Sadly, there appears to be only 1 USB controller in the system so i had to passthrough devices themselves.

-[0000:00]-+-00.0  Intel Corporation Skylake Host Bridge/DRAM Registers [8086:190f]
           +-01.0-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340] [1002:6613]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
           +-02.0  Intel Corporation HD Graphics 530 [8086:1912]
           +-14.0  Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f]
           +-14.2  Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131]
           +-16.0  Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a]
           +-17.0  Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102]
           +-1b.0-[02]--
           +-1c.0-[03]--
           +-1c.4-[04]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340] [1002:6613]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
           +-1d.0-[05]--
           +-1f.0  Intel Corporation Sunrise Point-H LPC Controller [8086:a144]
           +-1f.2  Intel Corporation Sunrise Point-H PMC [8086:a121]
           +-1f.3  Intel Corporation Sunrise Point-H HD Audio [8086:a170]
           +-1f.4  Intel Corporation Sunrise Point-H SMBus [8086:a123]
           \-1f.6  Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]

01:00.0 0300: 1002:6613 (rev 87)
        Subsystem: 1462:8094
        Kernel driver in use: vfio-pci
        Kernel modules: radeon
01:00.1 0403: 1002:aab0
        Subsystem: 1462:aab0
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
04:00.0 0300: 1002:6613 (rev 87)
        Subsystem: 1462:8094
        Kernel driver in use: vfio-pci
        Kernel modules: radeon
04:00.1 0403: 1002:aab0
        Subsystem: 1462:aab0
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

vfio-pci is a driver that is used for passing things to a linux guest in an extremely efficient manner.

I include the radeon driver in /etc/initramfs-tools/modules
I blacklist the radeon driver in /etc/modprobe.d/blacklist.conf

i downloaded the latest ovmf from git and modified /etc/libvirt/qemu.conf to have

nvram = [
        "/srv/ovmf-x64/OVMF_CODE-pure-efi.fd:/srv/ovmf-x64/OVMF_VARS-pure-efi.fd"
]

my grub commandline is as follows:

GRUB_CMDLINE_LINUX_DEFAULT="rd.driver.pre=vfio-pci quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt"

There are a few items to add in here since i also have a 915 for the onboard skylake video, but not including those commands had no negatives that I've seen as of yet.

i load into /etc/modules the following

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
kvm
kvm_intel

/etc/vfio-pci1.cfg

0000:01:00.0
0000:01:00.1

/etc/vfio-pci2.cfg

0000:04:00.0
0000:04:00.1

My /etc/network/interfaces is as such:

auto br0
iface br0 inet dhcp
  bridge_ports enp0s31f6
  bridge_stp off
  bridge_maxwait 0
  bridge_fd 0
  post-up ip link set dev enp0s31f6 mtu 9000

in /etc/udev/rules.d/70-persistent-net.rules i have

SUBSYSTEM=="net", ACTION=="add", KERNEL=="tap*", ATTR{mtu}="9000"

my launch script for system 1 is

#!/bin/bash

configfile=/etc/vfio-pci1.cfg
vmname="system1"

vfiobind() {
   dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}


if ps -A | grep -q $vmname; then
   echo "$vmname is already running."
   exit 1

else

   cat $configfile | while read line;do
   echo $line | grep ^# >/dev/null 2>&1 && continue
      vfiobind $line
   done


if [ ! -f /tmp/system1.fd ]; then
  cp /srv/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/system1.fd
fi

taskset -c 0,3 qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -serial none \
  -parallel none \
  -machine type=q35,accel=kvm \
  -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
  -smp 2,sockets=1,cores=1,threads=2 \
  -enable-kvm \
  -m 3G \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -usb -usbdevice host:040b:2000 -usbdevice host:046d:c01e -usbdevice host:0d8c:0014 -usbdevice host:046d:0819 \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom1.rom \
  -device vfio-pci,host=01:00.1,bus=root.1 \
  -drive if=pflash,format=raw,readonly,file=/srv/ovmf-x64/OVMF_CODE-pure-efi.fd \
  -drive if=pflash,format=raw,file=/tmp/system1.fd \
  -boot order=c \
  -device virtio-scsi-pci,id=scsi \
  -drive file=/dev/sdc1,id=disk0,if=none,format=raw,cache=none,aio=native \
  -device scsi-hd,drive=disk0 \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=52:54:00:ea:c8:8d

#  -mem-path /dev/hugepages1 \

exit 0
fi

system2 is 

#!/bin/bash

configfile=/etc/vfio-pci2.cfg
vmname="system2"

vfiobind() {
   dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}


if ps -A | grep -q $vmname; then
   echo "$vmname is already running."
   exit 1

else

   cat $configfile | while read line;do
   echo $line | grep ^# >/dev/null 2>&1 && continue
      vfiobind $line
   done


if [ ! -f /tmp/system2.fd ]; then
  cp /srv/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/system2.fd
fi

taskset -c 1,2 qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -serial none \
  -parallel none \
  -machine type=q35,accel=kvm \
  -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
  -smp 2,sockets=1,cores=1,threads=2 \
  -enable-kvm \
  -m 3G \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -usb -usbdevice host:04f2:0939 -usbdevice host:04f2:0833 \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom2.rom \
  -device vfio-pci,host=04:00.1,bus=root.1 \
  -drive if=pflash,format=raw,readonly,file=/srv/ovmf-x64/OVMF_CODE-pure-efi.fd \
  -drive if=pflash,format=raw,file=/tmp/system2.fd \
  -boot order=c \
  -device virtio-scsi-pci,id=scsi \
  -drive file=/dev/sdd1,id=disk0,if=none,format=raw,cache=none,aio=native \
  -device scsi-hd,drive=disk0 \
  -netdev type=tap,id=net0,ifname=tap1,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=52:54:00:ea:c8:8e

#  -mem-path /dev/hugepages2 \

exit 0
fi

i create a screen session for each system and launch from within screen. This gives me the qemu console so i can wake the system up from sleep and such if needed.

I plan to reintroduce hugepages once i get the extra memory installed, although the guide here doesnt go into detail on using hugepages for 2 guests, you can simply add it to fstab from what i can tell and point each system to its relevant hugepage mount point.

 

Obviously, with projects come issues, here are my issues, resolved or otherwise.

 

 

---

Issue1: Resolved!

that all said, the second system won't use the video card. i deleted the vga none line and built the whole system in qemu and in windows, i dont even see the video card listed in any state. It seems like the pci passthrough is being completely ignored. the strange thing is, it worked once and never again. Any thoughts come to mind?

got my second system issues resolved (rebooted host...)
 

---

Issue2:
anyway, both systems are running great, my windows score on each is a 5.1. each system has 3G ram, 1core and 2threads (pinned) and a radeon r7 240 3G ddr3. I pass through a usb audio device, mouse, and keyboard to each system. for disk, each system has an SSD passed through except windows detects it as a thin provisioned disk (how do i fix this???)

I passed the SSD through as such: 

 

 

  -device virtio-scsi-pci,id=scsi \
  -drive file=/dev/sdc1,id=disk0,if=none,format=raw,cache=none,aio=native \
  -device scsi-hd,drive=disk0 \

and

  -device virtio-scsi-pci,id=scsi \
  -drive file=/dev/sdd1,id=disk0,if=none,format=raw,cache=none,aio=native \
  -device scsi-hd,drive=disk0 \

for each machine respectively. I'm assuming i did something wrong here, but setting up lvm didn't interest me as I dont need snapshots or growth potential. 

---

Issue3: Resolved!

My next issue (which really doesnt matter much) is MTU

the host nic is 9000MTU, the br0 is 9000MTU, until a tap device joins, then it drops to 1500. I'm not sure how to get the tap device MTU set, but i'm actively researching it.

using this guide: 
https://linuxaleph.blogspot.com/2013/01/how-to-network-jumbo-frames-to-kvm-guest.html

i manually resolved the issue, kid's are on their VMs so i cant test persistence yet.

---

Issue4:
another issue is a slight crackle in the audio, no delays.. just crackly. im passing through the usb device via

-usb -usbdevice host:046d:0819

I'm still researching this particular issue.

---

Issue5:
aside from all those, i've trained my kids to put their computers to sleep when they finish, which they do... to my dismay since they can't wake the system up! i have to go into qemu and

sendkeys ctrl 100

a couple times and it wakes up. I checked the ovmf config to see if there was a mouse wakeup event, but i found none.

[cookie eater 965]

ok cool, i dont know what the bottom half of your post means, but good job!