Create a CI Build
Now that we have everything running locally we want to create a CI process to build our Wheel, publish that as an artefact and of course to test our code.
In the root of the project is a file called azure-pipelines.yml. This Yaml file describes the build process that takes place.
We will step through the file here to understand what is happening.
First we target a Ubuntu agent, this will work on Windows but will be horribly slow as installing the dependencies on Windows is very slow. As Databricks uses Python 3.5 we are setting a variable to force the use of this version specifically.
This is a PowerShell core to configure Databricks-Connect by inserting variables (that you must define in your build) into the config file. It also copies the Int environment config file to the default config.json as this is the configure we want to use in our tests. We can still deploy this build to other environments later.
The above step copies the Databricks-Connect config file to the root of the agent users home directory as this is a system default location to look for the file.
We now install the dependencies and execute our pytest unit tests. Note that we are outputting the test results in junit format which Azure Devops recognises and can publish.
Before we build the Wheel we need to set the build number in the root init file and in the setup.py files. This ensures we can track what version of code we are running later.
Lastly we execute the same Build.ps1 script to copy the files needed. These will be published as the output artefact and available to download at the end of the build.
The output of the build with test results looks like this:
This build is available publicly to see the results here: https://dev.azure.com/datathirst/Public-Demos/_build?definitionId=37&_a=summary