[FIXED] Best practices for Spring boot testing against authenticated remote system

Issue

I have written code that leverages Azure SDK for Blobs in order to interact with the blob storage.

As a clever and dutiful developer, I have not tested my code by navigating the live application, but rather created a Spring Boot JUnit test and spent a few hours fixing all my mistakes. I didn't use anyh kind of mocking, in fact, as my problem was using the library the correct way. I ran the code against a live instance of a blob storage and checked that all my Java methods worked as expected.

I am writing here because

• To call it a day, I hardcoded the credentials in my source files. The repository is a company-private repository, not that harm. Credentials can be rotated, developers can all access from Azure portal and get the credentials. But still I don't like the idea of pushing credentials into code
• Having these junit tests work on Azure DevOps pipelines could be some of a good idea

I know from the very beginning that hardcoding credentials into code is a worst practice, but since this morning I wanted to focus on my task. Now I want to adopt the best practices. I am asking about redesigning the test structure

Testing code is this.

The code creates an ephemeral container and tries to store/retrieve/delete blobs. It uses a GUID to create a unique private workspace, to clear after test is finished.

@SpringBootTest(classes = FileRepositoryServiceAzureBlobImplTest.class)
@SpringBootConfiguration
@TestConfiguration
@TestPropertySource(properties = {
        "azure-storage-container-name:amlcbackendjunit",
        "azure-storage-connection-string:[not going to post it on Stackoverflow before rotating it]"
})
class FileRepositoryServiceAzureBlobImplTest {

    private static final Resource LOREM_IPSUM = new ClassPathResource("loremipsum.txt", FileRepositoryServiceAzureBlobImplTest.class);
    private FileRepositoryServiceAzureBlobImpl uut;
    private BlobContainerClient blobContainerClient;
    private String loremChecksum;

    @Value("${azure-storage-connection-string}")
    private String azureConnectionString;
    @Value("${azure-storage-container-name}")
    private String azureContainerName;

    @BeforeEach
    void beforeEach() throws IOException {

        String containerName = azureContainerName + "-" + UUID.randomUUID();
        blobContainerClient = new BlobContainerClientBuilder()
                .httpLogOptions(new HttpLogOptions().setApplicationId("az-sp-sb-aml"))
                .clientOptions(new ClientOptions().setApplicationId("az-sp-sb-aml"))
                .connectionString(azureConnectionString)
                .containerName(containerName)
                .buildClient()
        ;


        blobContainerClient.create();
        uut = spy(new FileRepositoryServiceAzureBlobImpl(blobContainerClient));
        try (InputStream loremIpsumInputStream = LOREM_IPSUM.getInputStream();) {
            loremChecksum = DigestUtils.sha256Hex(loremIpsumInputStream);
        }

        blobContainerClient
                .getBlobClient("fox.txt")
                .upload(BinaryData.fromString("The quick brown fox jumps over the lazy dog"));

    }

    @AfterEach
    void afterEach() throws IOException {
        blobContainerClient
                .delete();
    }

    @Test
    void store_ok() {
        String desiredFileName = "loremIpsum.txt";


        FileItemDescriptor output = assertDoesNotThrow(() -> uut.store(LOREM_IPSUM, desiredFileName));
        assertAll(
                () -> assertThat(output, is(notNullValue())),
                () -> assertThat(output, hasProperty("uri", hasToString(Matchers.startsWith("azure-blob://")))),
                () -> assertThat(output, hasProperty("size", equalTo(LOREM_IPSUM.contentLength()))),
                () -> assertThat(output, hasProperty("checksum", equalTo(loremChecksum))),
                () -> {
                    String localPart = substringAfter(output.getUri().toString(), "azure-blob://");
                    assertAll(
                            () -> assertTrue(blobContainerClient.getBlobClient(localPart).exists())
                    );
                }
        );
    }

}

In production (but also in SIT/UAT), the real Spring Boot application will get the configuration from the Container environment, including the storage connection string. Yes, for this kind of test I could also avoid using Spring and @TestPropertySource, because I'm not leveraging any bean from the context.

Question

I want to ask how can I amend this test in order to

1. Decouple the connection string from code
2. Softly-ignore the test if for some reason the connection string is not present (e.g. developer downloaded the project the first time and wants to kick-start) (note 1)
3. Integrate this test (with a working connection string) from Azure DevOps pipelines, where I can configure virtually any environment variable and such

Here is the build job comprised of tests

          - task: Gradle@2
            displayName: Build with Gradle
            inputs:
              gradleWrapperFile: gradlew
              gradleOptions: -Xmx3072m $(gradleJavaProperties)
              options: -Pci=true -PbuildId=$(Build.BuildId) -PreleaseType=${{parameters.releaseType}}
              jdkVersionOption: 1.11
              jdkArchitectureOption: x64
              publishJUnitResults: true
              sqAnalysisEnabled: true
              sqGradlePluginVersionChoice: specify
              sqGradlePluginVersion: 3.2.0
              testResultsFiles: '$(System.DefaultWorkingDirectory)/build/test-results/**/TEST-*.xml'
              tasks: clean build

Note 1: the live application can be kick-started without the storage connection string. It falls back to a local temporary directory.

Solution

The answer is a bit complex to explain, so I did my best

TL;DR

Note that the original variable names are redacted and YMMV if you try to recreate the example with the exact keys I used

• Create a secret pipeline variable containing the connection string, and bury* it into the pipeline

Example name testStorageAccountConnectionString

• Change the Gradle task

              - task: Gradle@3
                displayName: Build with Gradle
                inputs:
                  gradleWrapperFile: gradlew
                  gradleOptions: -Xmx10240m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -DAZURE_STORAGE_CONNECTION_STRING=$(AZURE_STORAGE_CONNECTION_STRING)
                  options: --build-cache -Pci=true -PgitCommitId=$(Build.SourceVersion) -PbuildId=$(Build.BuildId) -Preckon.stage=${{parameters.versionStage}} -Preckon.scope=${{parameters.versionScope}}
                  jdkVersionOption: 1.11
                  jdkArchitectureOption: x64
                  publishJUnitResults: true
                  sqAnalysisEnabled: true
                  sqGradlePluginVersionChoice: specify
                  sqGradlePluginVersion: 3.2.0
                  testResultsFiles: '$(System.DefaultWorkingDirectory)/build/test-results/**/TEST-*.xml'
                  tasks: clean build
                env:
                 AZURE_STORAGE_CONNECTION_STRING: $(testStorageAccountConnectionString)

Explanation

• Spring Boot accepts placeholder ${azure.storageConnectionString} from an environment variable AZURE_STORAGE_CONNECTION_STRING. Please read the docs and try it locally first. This means we need to run the test with an environment variable propely set in order to resolve the placeholder
• Gradle can run with -D to add an environment variable. -DAZURE_STORAGE_CONNECTION_STRING=$(AZURE_STORAGE_CONNECTION_STRING) adds an environment variable AZURE_STORAGE_CONNECTION_STRING to the test run equal to the pipeline environment variable AZURE_STORAGE_CONNECTION_STRING (not that fantasy)
• Azure DevOps pipelines protect secret variables from unwanted access. We created the pipeline variable as secret, so there is another trick to do first

Gradle's env attributes set environment variable for the pipeline container. In this case, we make sure that Gradle runs with AZURE_STORAGE_CONNECTION_STRING set to testStorageAccountConnectionString. Env is the only place where Azure pipelines agent will resolve and set free the content of the secret variable

• Secrets cannot be retrieved any more from web interface. Azure Pipelines are designed for this

Answered By - usr-local-ΕΨΗΕΛΩΝ
Answer Checked By - Marilyn (JavaFixing Volunteer)