[FIXED] Windows: Automatically Detect, Stripe, Provision, Mount AWS EC2 Ephemeral Disks for Jenkins Agents

Issue

I have windows build agents for Jenkins running in EC2, and I would like to make use of the ephemeral disks coming from the "d" type instances (C5ad.4xl for example gives 2 x 300GB NVMe) to take advantage of the high IO available on those disks.

Since they are build agents the ephemeral nature of the drives is fine. I need something that will detect, provision and mount those disks as a drive in Windows basically regardless of size and number. I can do this easily (LVM or software RAID etc.) in Linux but although there is a guide from 2014 here for achieving this, it doesn't seem to work on Windows Server 2019 and the latest instances.

That same post makes reference to new commandlets added from Server 2012 R2 but those do not support converting the disks to dynamic (which is a key step needed to stripe them done by diskpart in the original post's code), so they cannot be used to directly do what is required.

Are there any other options to make this work dynamically, ideally with powershell (or similar) that can be passed to the Jenkins agent at boot time as part of its config?

Solution

Windows now has Storage Pools and these can be used to do what is needed here. This code successfully detected multiple disks, added them to the pool striped, used the max size available and mounted the new volume on the drive letter "E":

# get a list of the disks that can be pooled
$PhysicalDisks = (Get-PhysicalDisk -CanPool $true)
# only take action if there actually are disks
if ($PhysicalDisks) {
    # create storage pool using the discovered disks, called ephemeral in the standard subsystem
    New-StoragePool –FriendlyName ephemeral -StorageSubSystemFriendlyName "Windows Storage*" –PhysicalDisks $PhysicalDisks
    # Create a virtual disk, striped (simple resiliency in its terms), use all space
    New-VirtualDisk -StoragePoolFriendlyName "ephemeral" -FriendlyName "stripedephemeral" -ResiliencySettingName Simple -UseMaximumSize
    # initialise the disk
    Get-VirtualDisk -FriendlyName 'stripedephemeral'|Initialize-Disk -PartitionStyle GPT -PassThru
    # create a partition, use all available size (this will pop up if you do it interactively to format the drive, not a problem when running as userdata via Jenkins config)
    New-Partition -DiskNumber 3 -DriveLetter 'E' -UseMaximumSize
    # format as NTFS to make it useable
    Format-Volume -DriveLetter E -FileSystem NTFS -Confirm:$false
    # this creates a folder on the drive to use as the workspace by the agent
    New-Item -ItemType Directory -Force -Path E:\jenkins\workspace
}

There are some assumptions here about the number of disks, and that will vary based on the instance type, but generally it will take any ephemeral disks it finds, if there is more than one it will stripe across them, and then uses the entire disk size available once the volume has been created/formatted. This can all be wrapped in <powershell></powershell> and added to the user data section of the Jenkins agent config so that it is run at boot.

Answered By - Adam Comerford