Category: Servers & Services

I posted a couple of weeks back that I was soon to be the recipient of a new gigabit symmetric link to the intertubes – and there was a bit of a caveat of their use of CG-NAT due to the exhaustion of the IPv4 address space.

To compensate I’ve routed everything over a VPN link in prep for this new connection, which does work fantastically well.

After a chat with their support guys – and I did actually manage to speak to someone in their Networks team, not just a customer support droid(!), they will soon be introducing a service where I can add a public IPv4 to my account, so I’ll be able to run at full gigabit speeds without the 100Mbit restriction of this VPN connection!

I’m told that this will be within the next couple of months, which is around the same lead time as the connection itself, so hopefully this will be live just in time for my new fibre.

More updates to come as this progresses!

Everyone who reads this website will know about the cost of living crisis – this is a global problem brought about by the combination of the Covid-19 pandemic, and the unfortunate situation in Ukraine at the present time.

For us in the UK, the costs of energy have skyrocketed in recent months, with our government permitting the energy companies to charge us all an extra £600 per year in electricity alone. Given I run servers, this isn’t particularly good news.

The cost of running the network infrastructure behind this blog is not insignificant – just the server alone runs me around £30-£40 per month just in power, and add onto that the £45 in connectivity costs. I don’t meter the power taken by the router, switch etc, so the total cost will be marginally higher.
Given the increases that have hit, I expect to see a 3x increase in power costs to run this blog on the current hardware – an unsustainable jump.

I’ve got a couple of Raspberry Pi’s floating around, at the moment uncommitted to any tasks, so I thought I’d use one of these. A Pi CM4 module in a support PCB with dual network links should so just fine. I ogirinally got this to build a Pi based router (what it was designed for) – nevermind!

I’m using the new 64-Bit version of Raspberry Pi OS, and a 128GB MicroSD card for storage, since this board doesn’t have an open PCIe lane for a NVMe drive.

A quick setup with Nginx, php-fpm & MariaDB & I’m in business! The site is now running on a Raspberry Pi! As a plus, everything does seem to be quicker than the old Apache setup too. I have for now dropped a few features of the site, just until I get a feel of how quick this new setup is going to be, but they should be back soon!

For 90% of the time I’ve run this website, it’s been run from a server in my house, over a domestic-grade ADSL connection. The problem has always been the very asymmetric download/upload bandwidth provided by domestic ISPs. Normally it’s about a 10:1 ratio these days (the current connection is 350MBit DL / 20MBit UL, which is even worse). Going back a few years, this wasn’t much of an issue – 99.9% of general ISP customers didn’t need a high upload rate, as they were only requesting web addresses, etc. Nowadays especially with the rise of online video services & social media, upload rates are more important than ever, since a good proportion of the population is uploading their own content daily.

Despite these changes in how the networks are utilised by the end users, the upload rates haven’t really improved much, which I’ve always found baffling.

Considering the bandwidth limitations imposed by such a connection on upload rates, the site’s done pretty well over the years, even though it can be a touch slow at times. I’ve been looking at changing this situation for a long time, watching to see if any of the direct fibre ISPs in the UK were going to get around to serving my area, such as Gigafibre, CityFibre or HyperOptic, but they unfortunately haven’t yet appeared anywhere near my locale.

Cue the entry of a new ISP: Brsk. These guys are building an entirely new fibre network, and just so happen to be doing the network build in my direct area at the moment. For only £5 more a month than I’m paying for the service I have now, I can now obtain full gigabit symmetric connectivity!

There’s one small caveat to the service, which wouldn’t bother the average member of the public, is a serious block to how I use my internet connectivity – they don’t hand out globally addressable IPv4 addresses. This is ostensibly due to the global IPv4 address shortage. Instead, they use CGNAT to address their endpoints, which renders me unable to route anything out to the global internet from my connection. Luckily there are ways around this issue.

I do have access to some public IPv4 addresses that are currently unallocated, in another networking installation, which also runs a VPN into my home network. So it’s just a matter of hooking up my primary server to that VPN, and routing one of the spare IPv4s over the private link from the router at the other site. This will limit thing somewhat, as the other site link is a 100MBit symmetric leased-line, however that’s still an 80MBit improvement in upload rate overall.

This latter solution of routing a public IP over VPN to a private endpoint is something I’ve already done in preparation – and it works great. The firewall on the site router provides extra protection for the server, and even with the VPN overhead it’s just as quick as it was before.

Since they’re still building out the network, I’ve only been able to place a preorder, and the connection will be installed (allegedly) sometime in July. Once the new link is in, I can get rid of my existing provider.

Once the link’s installed I’ll provide some updates!

So, it’s time to finish off the upgrades to the core storage server on my network. Now a new motherboard, CPU & RAM have been obtained (MSI GA-X58-USB3), Core i7 950, 12GB), along with new SAS/SATA HBAs for the disk rack I can get everything fitted into place.

Proper branded LSI HBA cards are expensive so I went with the cheaper option & obtained a pair of Dell H200 RAID cards. These have custom firmware flashed to them, but luckily can be crossflashed to a standard LSI firmware to become an LSI9211-8i card – providing 8 lanes of either SAS or SATA connectivity on a pair of SFF-8087 ports. Flashing these cards was very simple, once I managed to work my way into the EFI shell on my main machine, which I was using to do the flashing. Find all the firmware files & required software here:

Dell PERC H200 Crossflash Files

1 file(s) 8.18 MB

Download

One thing I left out from the flashing was a BIOS – this means that the boot process is speeded up, but also means the system BIOS cannot see the disks connected to the cards, so they’re not bootable. This isn’t a problem however, as I never plan on booting from the data storage disk array.

The SAS2008 RoC (RAID on Chip) on these cards runs at around 8.5W thermal power, so some active cooling is required to keep temperatures within check. I have attached a 40mm fan to each card’s factory heatsink, using M3x25mm screws. Getting the screws to grab the heatsink was the tricky bit – I needed to crimp the outer corners of the fins together slightly, so when the screws are driven in, the gap is forced to expand, which grabs the threads. The fans will be connected to spare headers on the motherboard for speed monitoring.

It was a struggle finding a motherboard with the required number of high-lane-count PCIe slots. Even on modern motherboards, there aren’t many about within a reasonable price range that have more than a single x16 slot, and since I’m going with the new HBAs, a single slot is no longer enough. The motherboard I managed to obtain has a pair of x16 slots, and a x4 slot (x16 physical), along with a 3 x1 slots. The only downside is there’s no onboard graphics on this motherboard, so an external card will be required. Another cheapie from eBay sorted this issue out.

Since I need to use the x16 ports for the disk controllers, this card will have to go into the x4 slot.

Here the board has been installed into the new chassis, along with it’s IO shield. Both HBA cards are jacked into the x16 slots, with the SAS/SATA loom cables attached. I did have to grab longer cables – the originals I had were only 500mm, definitely not long enough to reach the ports on these cards, so 1m cables are used. The fans are plugged in with extensions to a pair of the headers on the motherboard, but the MB doesn’t seem to want to read RPM from those fans. Nevermind. While the fans are a little close to the adjacent cards, the heatsinks run just about warm to the touch, so there’s definitely enough airflow – not forgetting the trio of 120mm fans in the bulkhead just out of shot, creating a breeze right through the chassis.

Since the onboard SATA ports are in a better position, I was able to attach the boot SSD to the caddy properly, which helps tidy things up a bit. These slot into the 5-¾” bays on the front of the chassis, above the disk cage.

To take up the excess cable length, and tidy things up, the data loom to the disk cage is cable-tied to self-adhesive saddles on the side of the chassis. This arrangement also helps cooling air flow.

With the new components, and the cabling tied up, things inside the chassis look much cleaner. I’ve rationalised the power cabling to the disk backplanes down to a a pair of SATA power looms.

So I figured it was time to get a hardware update sorted for my network’s core storage server, which I have posted about before. The way I had the drives anchored to steel rails really doesn’t make moving or replacing disks easy, so a proper case needed to be sourced.

ServerCaseUK stocked 16-bay 4U chassis units, so one of these was ordered. These have 4 internal backplanes, with SFF-8087 Mini-SAS connections, so hooking into my existing 16-channel HBA card would be simple. In the current setup, the multi-lane cables are routed out via SFF-8088 connectors to the drive array, so this will tidy things up considerably.

The main data links are via these SFF-8087 connectors, each carrying 4 lanes of SATA.

Power is provided by 4x Molex connections, via SATA power adaptors (the good kind, which don’t create fire). There’s a 5th Molex hidden down the size of the last fan, which powers all 3 120mm fans.

The disks are kept cool by 3x 120mm hot-swap fans on the dividing wall. These don’t create much noise, and are always at full speed.

Here’s the back of the case after transplanting the motherboard & HBA from the old chassis. There’s a new 750W EVGA modular power supply, since I’ll be expanding the disk array as well. The boot SSD is currently sat on the bottom of the case since I don’t have a data cable long enough to mount it in the proper place as yet.

Here’s the fan controller, which takes care of the dual high speed Delta fans on the back wall of the chassis. This has a pair of temperature sensors – one on the HBA card’s heatsink, and the other on the fan wall monitoring the exhaust air temp of the drive array, to control the speed of the two fans. Temperatures are kept at around 30°C at all times.

Since the HBA card’s fan failed a while back, it’s had a couple of fans attached. The centrifugal one here works a little better than a massive 80mm axial fan, and is a little quieter. This is always run at full speed from a spare motherboard header. The temperature sensor feeding the fan controller can be seen here bonded to the heatsink. The 4 SFF-8087 cables are going off to the disk backplanes.

As mentioned before, there are a pair of 80mm Delta high-speed fans on the back wall of the case, to provide some extra cooling air flow just in case overheating manages to set in. These are usually spooled down to low RPM to keep them quiet.

Since space was getting a little tight, and having some slots spare on the HBA, I decided to add some more disks to bring the active members up to 12, from 9 – increasing available disk space from 28TB to 40TB.

/dev/md0:
        Version : 1.2
  Creation Time : Wed Mar 11 16:01:01 2015
     Raid Level : raid6
     Array Size : 39068874880 (37258.98 GiB 40006.53 GB)
  Used Dev Size : 3906887488 (3725.90 GiB 4000.65 GB)
   Raid Devices : 12
  Total Devices : 13
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Nov 18 14:13:35 2019
          State : active 
 Active Devices : 12
Working Devices : 13
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           Name : Main-PC:0
           UUID : 266632b8:2a8a3dd3:33ce0366:0b35fad9
         Events : 1653174

    Number   Major   Minor   RaidDevice State
      16       8      144        0      active sync   /dev/sdj
      11       8      160        1      active sync   /dev/sdk
       9       8      176        2      active sync   /dev/sdl
      10       8      208        3      active sync   /dev/sdn
      15       8      192        4      active sync   /dev/sdm
       5       8      112        5      active sync   /dev/sdh
       6       8       80        6      active sync   /dev/sdf
      13       8       96        7      active sync   /dev/sdg
       8       8       64        8      active sync   /dev/sde
      19       8        0        9      active sync   /dev/sda
      18       8       48       10      active sync   /dev/sdd
      17       8      128       11      active sync   /dev/sdi

      12       8       16        -      spare   /dev/sdb

The space expansion improves things there, I will be adding a couple more spare disks to bring the number up to the full 16, just in case of any failures.

There are still a couple of issues with this setup:

• The motherboard & CPU are ancient. Currently an Intel Core 2 Quad, running 8GB of RAM, limits data throughput, and critically, the speed of mdadm data checks & rebuilds. The Core 2 Quad also runs at roughly the same temperature as the Sun’s core when under high load.
• The SATA HBA is running 4 controllers on an expander, through a PCIe x4 link, which is a little slow due to congestion on the expander itself. RAID6 does have some write-speed penalties though.

These are issues I will address shortly, with a replacement motherboard on the way!

Just a quick note for the regular (and not so regular) readers:

My server infrastructure is currently going through some migration & upgrade processes, so for the next few days the site might be a little on the slow side, but hopefully I can get everything sorted out quickly!

During this time please bear with the slower than normal response times.

73s,

2E0GXE