Yesterday saw the Spark Summit keynote take place in San Francisco. There were quite a few announcements, some expected, some not so much. In order of importance to how I see things today this is my summary and view of them.Read More
Having recently tried to get DBConnect working on a Windows 10 machine I’ve realised things are not as easy as you might think.Read More
I’ve been waiting for this feature for what feels like forever.
Databricks-Connect is here! Well almost - it’s still preview, but the release looks imminent. It allows you to develop using an IDE like VSCode, PyCharm, IntelliJ etc and connect to a remote Databricks cluster to execute the task.Read More
Databricks-Connect is the feature I’ve been waiting for. It is a complete game changer for developing data pipelines - previously you could develop locally using Spark but that meant you couldn’t get all the nice Databricks runtime features - like Delta, DBUtils etc.Read More