Hi everybody, I want to share some experiences I had last couple of weeks using the ISO offered by Veeam to create a fulll immutable Linux repository in a blink of an eye.
This ISO was released to the world this year end of May during VeeamON in Miami. I attended a session hosted by Rick Vanover, Christoph Meyer and Hannes Kasparic about Hardened Repositories.
Read all about the release in the article of Stijn Marivoet aka Mr. VeeamClick
Introduction: what's a hardened repo ?
A hardenend Linux repository is a way to configure a Linux distribution you like as a backup target, strip it down to the bare essentials to keep the attack surface as small as possible and elevate the immutable flag of the file system. This flag prevent changing or deleting files on this system. Another advantage of the XFS filesystem is the fast-block-cloning mechanism wich enables the creation of very fast full synthetic backups.
If configured well, this gives you a very secured box which acts as a very stable and resilient backup repository and offers a good protection against ransomware at a very limited cost. You can use almost any type of hardware altough, always pay attention to RAID configurations and a high speed link...
Introduction: manual steps in hardening
When you want to setup such a hardened repo, you can do all the steps yourself. Most engineers I've talked with are installing Linux themselves (some like Ubuntu, others Debian, or even Suicide Linux....) and then apply some type of hardening script.
Setting up a Linux machine is not that hard, but if you're working day-in day out on Windows machines, it can be challenging to configure networking, drives, partitions,... and in the end you must be sure you applied all harding requirements to secure the box as much as possible when this is becomes a production worthly machine.
On the VeeamHub (github repo) you can find an hardening script especially written for Ubuntu 20.04 based on DISA STIG.
Wat was shown in VeeamOn ?
During the hardened repo session with Hannes, Rick and Christoph they pleased the world with an all-in-one ISO. Creating a hardened remo will be easy as that. Write the ISO to a bootable stick and a script will repartition your target machine, handle networking, install Ubuntu 20.04,apply the DISA-STIG hardening and initialise the largest partition as backup repository.
Sounds really cool and is a time-saver every time you need to configure such a repo.
Last month I finally found some time to check this ISO.
As an engineer in the past I carried a lot of CD's, later DVD's, even more later bootable sticks with me with the most common ISO's to perform my job.
Typical these were some Windows Server ISO's, several flavours of ESXi, of course a recent VBR ISO and so on. That was reality unitl 4 years ago. I stumbled upon a device of IODD with a very nice capability. Dual mode: External Drive / Virtual CD-ROM drive
The IODD I have is the 2531 model and uses ISO files converted from CD, DVD, or Blu-ray
and presents them just like ordinary HDD or drive. It even can mount VHD files.
My model supports USB 3.0 with max 5Gbps transfer speeds
IODD, also available under the Zalman product line sells enclosures you can fit with a SSD of your own choice and then act as a portable drive. Nothing new here, but when you write your ISO's in the _ISO folder, you can mount immediatly select this ISO with the screen and button on the device and the drive then acts as an external DVD drive and let you boot from it.
It also has a mixed mode when you attach te device to a machine, you see the contents as an external drive and also the DVD drive with the ISO if you have mounted one.
Ideal for setting up barebone ESX and / or Windows, Linux machines, it saved me a lot of time.
IODD + VEEAM HARDENED REPO ISO: NO GOOD IDEA ?
So during my tests with the Hardened ISO, I just wrote the ISO to the device. Hooked it up to a physical server in my lab and booted from the ISO. It worked flawlessly and I immediatly entered the setup menu.
I worked my way through the setup. I had 2 disks in mirror (2x 240Gb) for the OS and a RAID5 set to perform some tests.
Setup is really easy and straight forward, entering some names, credentials, networkconfiguration and you're ready to go.
And then....
I disconnected my IODD device, rebooted the machine but it wasn't capable of finding a boot partition...
Strange... all went fine during setup, no errors,... must be a glitch in the matrix ?
In the meantime playtime was over and had to focus on other work. Now fate wanted me to need my IODD drive to install a fresh Windows server. As soon as I plugged in my ISO-library-on-the go, I got the following message: 1st Partition: EE. 🙀
No virtual CD-ROM visible, no external harddrive, no files, no ISO, just nothing.
A quick search on this errorcode guided me to the conclusion something very bad was wrong with my partition table.
Playing around with Minitool Partition Wizard I've found out that my drive had now a blank unformatted GPT partition... ouch, no more data.
Luckely with some other recovery tools (PM me if you want their exact names) I was able to recover all my data and ISO's to another external drive.
To be sure this was really caused by the Hardened ISO, I reformatted the IODD, put again the ISO and mounted it to the same machine.
I went again through all steps and the result was exaclty the same.
So be cautious when you want to use this ISO on a device which is capable of simulating a virtual drive of an ISO that resided on the disk. You could end up with an unreadable bricked external drive.
There is a warning is during install:
The ISO will automatically re-format your disk storage; the smallest volume will be used for the OS, the volume for the backup files.
Fine for me, but not on external mounted drives please.....
Definitly something I'll adress to Hannes and his team, but for now, pay attention when you're using such hybrid virtual-DVD-ROM drives together with the Veeam Harderned Repository ISO.
