Setting up shop – The MLOps way

Getting Allure up and running

Allure is an open-source reporting tool that consumes automated test output and presents it to give a "test suite"-like overview.

First an example

Let us have a look at an existing report.

This one is generated on every development CI run.

MLOps Allure report

Installing Allure

Allure for report generation is not currently offered through brew or pip or any other package managers, so we have to install it manually. Allure-behave is available.

The same goes for installing it as part of pipeline execution.

The installation consists of two parts:

allure-behave
- (the formatter)
allure
- (the report generator)

Installing it all via an ADO script task

parameters.allure_behave_version = 2.12.0

parameters.allure_version = 2.20.1

# Install Allure and dependencies
- script: |
    python -m pip install allure-behave==${{ parameters.allure_behave_version }}
    curl -o allure-${{ parameters.allure_version }}.tgz -Ls https://github.com/allure-framework/allure2/releases/download/${{ parameters.allure_version }}/allure-${{ parameters.allure_version }}.tgz   
    sudo tar -zxvf allure-${{ parameters.allure_version }}.tgz -C /opt/   
    sudo ln -s /opt/allure-${{ parameters.allure_version }}/bin/allure /usr/bin/allure
    allure --version
displayName: Install Allure and dependencies
workingDirectory: ./

Generate test results in the Allure format

Before Allure can use test results to create a report, the test results need to be generated and formatted for Allure use. This is done by the allure-behave formatter.

Run tests like this:

python -W ignore -m 
  behave 
  -f allure_behave.formatter:AllureFormatter 
  -o reports
  <path to feature files directory>

Here is a variation where specific tags are listed:

python -W ignore -m 
  behave 
  --tags '~@InProgress'
  -f allure_behave.formatter:AllureFormatter 
  -o reports
  <path to feature files directory>

Putting a ~ in front of a tag means you are omitting tests with that tag specifically.

See here for more examples.

Generate the Allure report (HTML)

# Generate Allure BDD report
- script: |
    allure generate -c reports -o reports/html
displayName: Generate Allure report
workingDirectory: ./

Running it locally is also possible

allure open reports/html

Populating the trend chart

### <run the tests here> ###

- task: AWSShellScript@1
displayName: Download existing Allure report history from S3
inputs:
    awsCredentials: ${{ variables.aws_connection }}
    regionName: ${{ parameters.aws_region }}
    scriptType: inline
    inlineScript: |
    mkdir reports/history
    aws s3 sync s3://<your-bucket>/<reports>/history reports/history

### <generate the allure report here> ###

# Upload Allure BDD report HTML to the S3 bucket
- task: AWSShellScript@1
displayName: Upload Allure report to S3
inputs:
    awsCredentials: ${{ variables.aws_connection }}
    regionName: ${{ parameters.aws_region }}
    scriptType: inline
    inlineScript: |
    aws s3 sync reports/html s3://<your-bucket>/<reports>/

Defining custom filters

It is possible to define custom filters for the Allure report. This is done by creating a categories.json file and placing it in the root of the report folder.

A suggestion here is to create it elsewhere in the repo and copy it to the report folder before generating the report.

[
  {
    "name": "Ignored tests", 
    "matchedStatuses": ["skipped"] 
  },
  {
    "name": "Broken tests",
    "matchedStatuses": ["broken"]
  },
  {
    "name": "Failed tests",
    "matchedStatuses": ["failed"]
  },
  {
    "name": "Successful tests",
    "matchedStatuses": ["passed"]
  }
]

More advanced examples of filtering:

[
  {
    "name": "Skipped Tests By @Disabled - Definition",
    "matchedStatuses": ["skipped"],
    "messageRegex": ".*@Disabled.*"
  },
  {
    "name": "Broken Tests - Definition",
    "matchedStatuses": ["broken"],
    "traceRegex": ".*WebDriverException.*",
    "messageRegex": ".*unknown error.*"
  },
  {
    "name": "Test Defects - Definition",
    "matchedStatuses": ["broken", "failed"],
    "traceRegex": ".*NoSuchElementException.*",
    "messageRegex": ".*no such element.*"
  },
  {
    "name": "Test Defects - Definition",
    "matchedStatuses": ["broken", "failed"],
    "traceRegex": ".*IndexOutOfBoundsException.*",
    "messageRegex": "Index.*"
  },
  {
    "name": "Product Defects - Definition",
    "matchedStatuses": [ "failed" ],
    "traceRegex": ".*ElementShould.*",
    "messageRegex": ".*Element should.*"
  },
  {
    "name": "Product Defects - Definition",
    "matchedStatuses": [ "failed" ],
    "traceRegex": ".*checkCondition.*",
    "messageRegex": ".*should have.*"
  }
]

And to use them, they need to be copied into the test results folder, prior to generating the Allure HTML report.

# Copy Allure categories.json definition to the reports folder before generating the HTML report
- script: |
    cp allure/categories.json /reports
displayName: Set Allure report configuration

← Previous Next →