To BMC or not to BMC

Posted on by Mike McClay

When a firmware upgrade does not go to plan. I was trying to update the bmc firmware on one of the X470D4U motherboards, that are in my homelab servers, and it looked like it had succeeded. But after it rebooted there was no web interface available. Nor could I connect with ssh, the session just hung. :/

Thankfully others have been there before, and as its a homelab server its not that difficult to access the server hardware. After a bit of google-fu, I was able to see that I needed a freeedos formatted usb stick and the previous bmc firmware to flash to. Just in case you get bitten by this here are the steps.

  1. Download the socflash bundle.
  2. Get a suitable usb stick to wipe and install freedos.
  3. Download Rufus and Freedos (download the LiteUSB image)
  4. Use Rufus to install the FreeDos image to the usb stick.
  5. Drop the img file from the bmc firmware download and the socflash.EXE from the socflash bundle to the usb stick and eject the stick so its safe to remove.
  6. Ensure that your X470D4U is setup in the bios for Legacy boot. I used Legacy and UEFI.
  7. Connect the usb stick to the rear usb ports.
  8. Boot the board with keyboard and monitor connected, select F8 to launch the boot menu when the Bios options appear.
  9. Boot from the usb stick.
  10. Run socflash.exe if=X470D4~1.ima option=r (replace the X470D4~1.ima with whatever you named the firmware if you renamed it.)
  11. It should ask you to confirm and if you do then it should flash the bmc to whatever version you downloaded to the usb stick.

For me it was version 2.20.00, as that’s what it was at before this botched upgrade started. Once I had flashed to that version, I was able to connect again to the bmc and reset the settings to factory defaults. Then I ran the update to the new 03.02.00 version without any issues. A bit of a pain, but at least its recoverable.

Homelab 2024

Posted on by Mike McClay

The old homelab continues to evolve and the current version has changed enough from the last time I posted about it to cover the what and why of it.

Dream Machine Pro Max acting as the main router. This replaces the Asus routers that I’d been using before and is a significant step up in capabilities. The option to expand out the features, using the UniFi ecosystem, so that it can be a PVR and management pane for home security and access is of interest. As well as its ability to mange the UniFi range of WiFi equipment. At the moment I’m enjoying the much more mature views and insights it offers into my network traffic and the more in depth management of it. It’s a powerful beast with 10G throughput for routing and firewall capabilities. As well as featuring IDS/IPS, but there is a throughput penalty if those are active.

Mikrotik CRS309-1G-8S+ this is the main switch for the lab for fiber and DAC connections. Together with a NETGEAR XS508M for 10G over Ethernet, and a combination of them both allows all my lab equipment to connect at a minimum of 10G speeds. They are connected to each other via a 10G fiber link.

2x Whitebox home built servers. Powered by an AMD 5900X cpu, 128GB of ram, on an AsRock Rack X470D4U Motherboard. This is currently running on WMWare EXSI 7, but will most likely be moved over to Proxmox. Due to the death of VMug (VMware user group) post Broadcom takeover, killing off discounted access to licenses for professionals to use in their lab work.

Synology 1821+ NAS This provides the core storage for the Vmware lab. The VM’s are hosted on NFS datastores. This NAS is backed up using snapshot replication to my older 1819+, as well as making use of snapshots to protect the active file system from ransomware and accidental deletion of data. It also hosts various Docker containers on its Container Manager service, as well as hosting the DNS zones for the lab.

I’m less impressed with this NAS over time as Synology keep removing or hiding features that professionals in the industry use as well as home users.

Such as hiding SMART metrics for the drives, overly promoting, or mandating the use of, their own rebranded drives. Cheeping out by removing CODEC support for HEIC/H.265 which were supported when the units shipped. Which impacted paying customers for their add on Security product, as well as home users using the devices for their advertised purpose as a media server, etc.

While there are workarounds to address these issues it’s increasingly indicative of a mindset change in the company. When it reaches end of life I may look elsewhere or roll my own.

Don’t cross the streams

Posted on by Mike McClay

So, my attempt at enjoyment while dosed with the flu was nixed by my setup denying me access to my audio media on my NAS.

A dive into the problem showed that it was really two problems. So, if your NAS, Synology in my case, is no longer visible over DLNA then a reboot of your network switches may be the solution. It was the last thing I tried, probably due to being dosed with the flu. But it seems to have fixed the undesired blocking of the DLNA SSDP advertisement over my network and restored my ability to see my NAS on DLNA clients again.

The second problem was that the Synology NAS was not showing all of the music and occasionally nothing at all via the Media Server app. This turned out to be a problem with the indexing service on the NAS, and the solution to fixing it was to change the file type on the indexed folders. After a lengthy time, it completed the indexing, and the Media Server app was able to see and serve the content once more. I had previously tried to reindex, but it was only when I changed the File Type for the indexed Folder in the Indexing Service that it fixed this issue.

After solving these issues I could once more stream music to Foobar 2000 on my ipad, and attempt to chill out while waiting for my own AV to kick this flu virus’s ass.

Memories are forever?

Posted on by Mike McClay

It was Homelab upgrade time and I decided to max out one of my servers with the full 128GB. These ASRock Rack X470D4U boards have worked out incredibly well in the lab and continue to delight. In this case it was due them working so well with the AMD desktop CPUs, that I was able to get 128GB of DDR4 ECC ram working at 3200MHz without any problems on a Ryzen 2700 CPU.

While it’s not on the memory QVL list for this board, I can happily report that Kingston Server Premier KSM32ED8/32HC dimms will work just fine and let you get the most from your board, should you need it.

If you are wondering why so much ram? Well, I felt like taking VMware’s Cloud Foundation platform for a spin to see how it had matured, and, well, it’s a tiny bit memory hungry what with being a complete cloud hosting platform.
So, this upgrade was the answer and I should have no problem in installing it and getting it running in my homelab now. And I’ll have plenty of memory for whatever else I want to deploy on this server over its many years of service to come, as I can always upgrade that Ryzen 2700 to a 5000 series CPU and keep this investment viable for quite some time.

Raising Lazarus

Posted on by Mike McClay

Well, after five years of service my DS1515+ succumbed to the Intel C2000 series CPU bug and died.

As I like to silver lining the crap out of any cloud coming my way, I upgraded to an eight bay DS1819+ to give me more room to grow into, and more importantly have a working NAS again.

This left me with this DS1515+ as a paperweight and that was not going to work for me. So I ordered up some 100 ohm resistors and busted out the trusty soldering iron to implement this fix https://mariushosting.com/synology-how-to-fix-c2000-flaw-with-100-ohm-resistor/ .

A quick bit of cautious soldering later and the DS1515+ was alive again, populated with 10TB drives, setup with BTRFS (with snapshots) and now suitable to be a data replication target for the main NAS.

Every cloud, silver lining. 🙂

Good Vibrations

Posted on by Mike McClay
So, I was getting a little annoyed with my DS1515+ in that the fans were starting to age and creating a lot more noise.
A quick solution was to swap them out for Nocuta NF-A8 FLX fans and to use the rubber fan mounting grommets that come with it to attach the fans to the fan shrouds. These help absorb the fan vibrations and further lower the noise from the new fans.
An additional fix, for HD sled vibrations, was employed to further lower the overall noise from vibrations in the case.
For a tutorial on how to do this case mod see https://www.youtube.com/watch?v=z-aTvVo59tY it works by using the cushioning properties of the loops part of a strip of adhesive Velcro tape.
When the drive sleds are re-inserted into the case they rest on this and it absorbs their vibrations. The result is truly awesome if you were suffering from too much noise from the drives vibrating in their drive bays.

Pack mentality

Posted on by Mike McClay

Now that the lab is (partly) updated. I’ve been updating my packer + ansible builds to take advantage of this new capacity. There is something awesome about being able to run a build script and have it build out all the vm’s and then call ansible to install and config a Cloudera cluster + the data science workbench.

Waiting for the new AMD cards to launch so I can pick up a cheap CUDA card (assuming the NV prices drop) for TesnsorFlow offload.

Looking forward now to the Zen2 cpu’s releasing to market so that I can see what I’m doing with the rest of the lab. Will it be a 12core Ryzen, Gen3 Threadripper, or perhaps an Epyc based system that joins this one? Who knows, all I do know is that the cpu market is interesting again now that AMD are revitalised. I look forward to what real competition will bring from AMD and Intel now they are competing again.

Homelab upgrades

Posted on by Mike McClay

Time to upgrade my homelab. It was time for my old C2750D4I Avoton server to retire as it was getting too slow for my needs. Now that a re-emergent AMD has helped give the x86 market a much needed kick in the pants, there are a lot more options than when I last upgraded the lab.

The details are :-

Whitebox home built, Ryzen 2700 based, eight core (sixteen thread) server with 64gb of ram (ASRock X470D4U motherboard). Two 2TB SSD drives provide local storage options and I plan to chuck a 10GBE card and a CUDA card into it so I can model some “big data” solutions in my lab.

This is just the first phase of my home lab upgrades as Zen2 is going to come out in a month or so and I’m looking forward to upgrading some more of my servers to Ryzen 3000 series cpus or to the Zen2 based Threadripper cpus when they hit the market.

Importing existing VMFS datastores

Posted on by Mike McClay
So, one of my VMware homelab servers has decided that its built in storage ports are the work of the devil and it will no longer behave with the disks attached to them.

Handly enough I had a spare IBM M1015 HBA lying around and was able to stuff that in it and reconnect the storage to that.

So how to add the datastores back into VMware?

Thankfully that’s nice and easy.

ssh onto the VMware server, and get it to list mountable volumes that are not mounted with: –

>esxcfg-volume -l

Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).
VMFS UUID/label: 555ccf8d-1e83b310-841f-d050996fdfcc/MainStore
Can mount: Yes
Can resignature: Yes
Extent name: naa.50014ee2af9fd52b:1 range: 0 – 610303 (MB)

Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 58fa7c2e-4e0fb8d8-ceaf-d050996fdfcc/Datastore1
Can mount: Yes
Can resignature: Yes
Extent name: naa.5000cca249d38d3d:1 range: 0 – 3815423 (MB)

So, we can see the two lab datastores on the two disks.
Lets get those mounted and then we can get this host back into action.

> esxcfg-volume -M 555ccf8d-1e83b310-841f-d050996fdfcc
Mounting volume 555ccf8d-1e83b310-841f-d050996fdfcc
> esxcfg-volume -M 58fa7c2e-4e0fb8d8-ceaf-d050996fdfcc
Mounting volume 58fa7c2e-4e0fb8d8-ceaf-d050996fdfcc

That should be them back into action. Lets check.

>esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
————————————————- ———– ———————————– ——- —— ————– ————-
/vmfs/volumes/f16a5c2f-024e8e4b NFS f16a5c2f-024e8e4b true NFS 15857184727040 6747773542400
/vmfs/volumes/555ccf8d-1e83b310-841f-d050996fdfcc MainStore 555ccf8d-1e83b310-841f-d050996fdfcc true VMFS-5 639950127104 423815544832
/vmfs/volumes/58fa7c2e-4e0fb8d8-ceaf-d050996fdfcc Datastore1 58fa7c2e-4e0fb8d8-ceaf-d050996fdfcc true VMFS-6 4000762036224 3874391851008
/vmfs/volumes/9d5d25e5-72a247b9-fc58-db0a033ab4bf 9d5d25e5-72a247b9-fc58-db0a033ab4bf true vfat 261853184 107995136
/vmfs/volumes/4ea4f4e0-ce3c1495-0118-faa5b7f0c306 4ea4f4e0-ce3c1495-0118-faa5b7f0c306 true vfat 261853184 110907392
/vmfs/volumes/555b82b5-b0de9b80-885e-f46d043c4066 555b82b5-b0de9b80-885e-f46d043c4066 true vfat 299712512 83746816

As we can see above, the datastores MainStore and Datastore1 are back in action and we should be able to use the vm’s on it.

Mine (vm’s) were in perfect condition and I was able to get my homelab Cloudera Hadoop cluster backup and running and upgrade it to CDH 5.12.0 to play with.

The Vmware 5.5 u3 update waltz in esxcli minor

Posted on by Mike McClay

So update 3 for 5.5 is out and you want to deploy it with esxcli, follow these steps and you will be updated in no time.

0) esxcli vm process list , to make sure the host is empty.

1) vim-cmd /hostsvc/maintenance_mode_enter

2) esxcli network firewall ruleset set -e true -r httpClient

3) ~ # esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep “ESXi-5.5.0-201509”
ESXi-5.5.0-20150902001-standard   VMware, Inc.  PartnerSupported
ESXi-5.5.0-20150901001s-standard  VMware, Inc.  PartnerSupported
ESXi-5.5.0-20150901001s-no-tools  VMware, Inc.  PartnerSupported
ESXi-5.5.0-20150902001-no-tools   VMware, Inc.  PartnerSupported

If this shows nothing or errors out, then the chances are that you are behind a proxy for HTTP traffic.
So use the proxy option to pass that to the esxcli command.
esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml –proxy http://proxy.mycompany.com:8080  | grep “ESXi-5.5.0-201509”

4) ~ # esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.5.0-20150902001-standard

Like step 3, if this fails, then do the proxy shuffle and you should be able to pull down the patch and update.

5) reboot.

6) vim-cmd /hostsvc/maintenance_mode_exit

And your done, and should be seeing this when you query the version.

~ # esxcli system version get
Product: VMware ESXi
Version: 5.5.0
Build: Releasebuild-3029944
Update: 3