Azure Weekly

Issue #447 - 3rd December 2023

Let's start this week with AI; ML.NET 3.0 Boosts Deep Learning, Data Processing for .NET-Based AI Apps, there's also a nice Getting started with Azure AI Studio, and a feature I'm quite interested in trying - Write Your Git Commits with GitHub Copilot (hopefully it can deliver something more meaninful than "tweak", "change", "tweak" which too many of my commit messages seem to be).

In the data and analytics space: Semantic Link: OneLake integrated Semantic Models and Microsoft Fabric real-time analytics exploration:KQL Database mirroring and What are Kusto Query Language KQL databases in Microsoft Fabric?.

In Cloud Native Apps, a particularly good article - Getting Hands-On with Azure Functions: A Deep Dive and Secure access to your Azure App Services/Function Apps/Logic Apps using Azure AD OAuth and API.

I mentioned this topic earlier in the year, but code = energy = money = carbon. Creating highly efficient code, can have a huge positive impact on your infrastructure, your budget, and the environment. Several years ago we worked on a data and analytics project that needed to inject large amounts of real time data - marine shipping vessel locations from the entire planet. The initial implementation used a popular Python package (a wrapper over a C++ library) and that could process 50,000 AIS (Automatic Identification System) messages per second, and needed a 45 node cluster in order to ingest the data volume. We hypothesised that we could do a much better job, and re-implemented the AIS specification using C# / .NET Core 2.0 and achieved 2.6 million messages / sec / core, and allowed us to move the entire data ingestion process into a single Azure Function and reduced the ingestion lag from hours to near real time. With ever subsequent release of .NET we have seen significant performance improvements (with no additional code changes); and with .NET 8.0 was no exception; we've seen AIS.NET performance boosted by 27% which means we can now process 4.75 million messages / sec / core. Python may well be the most popular language for data science / engineering workloads, but for computationally intensive workloads, .NET is a great choice (and in my opinion needs better support in Spark & ML training environments like Microsoft Fabric).