Tech Talks

 Introduction  This blog is about the process of setting up SFTP server and then connecting to linked service in Azure Data Factory successfully. Prerequisites:- 1. You already have create a Linux machine from Azure Portal. 2. You have already created a data factory.  IMP Note:- **Throughout the demonstration to establish SFTP server & connecting to ADF, port 22 should be allowed as an inbound rule in NSG configured for linux vm. The flow of the blog is such that we will first setup a new user named ' sftpuser ' and restrict the permissions for access such that it is the only user for authentication to the SFTP server. Then we will connect to SFTP server via ADF Linked Service.  Unless explicitly stated otherwise in a step, all commands are to be executed by the user account created during the Linux machine deployment. Step-By-Step Process:- 1. Login to your virtual machine I am using gitbash, you can use any external tool as well such as Putty. The below command...

This blog is going to be an interesting one — we’re diving into how to connect to a NoSQL resource from Azure Synapse pipelines using a Managed Virtual Network (VNet) , and how to securely access Azure Cosmos DB for NoSQL via a private endpoint . What makes this post exciting is that we’re not just talking about assigning the Synapse workspace’s managed identity to Cosmos DB with Contributor or Reader access and calling it a day. Nope — we’re going deeper! We’ll explore: What is  Data Exfiltration Protection (DEP)? What’s really happening behind the scenes? The difference between the control plane and data plane And how all of this works in Azure Data Factory (ADF) Alright, let’s get started!   Here’s a simple diagram showing the overall architecture of a DEP-enabled Synapse workspace . In this setup, pipelines are created within Synapse, and the linked service connects to Azure Cosmos DB for NoSQL using a Managed VNet Integration Runtime via a priv...

 Introduction This blog post aims to demonstrate how you can configure an alert system for your ADF pipelines without relying on the in-build alert rule system provided by Azure Data Factory. But you may wonder why do i need to do that when I already have a in-build alert rule system managed by azure monitor for us? The answer is very simple. The manual create an alert system within the pipeline can help in following situations:- 1. Lets say, you are a part of a distributed email group that gets an alert when pipeline fails. Due to unusual circumstances at some point in time if the pipeline failure doesn't trigger alert email then it becomes a critical issue as some pipelines may belong to production environment and could certainly impact the business. The problem can be isolated by testing whether distributed email group is unresponsive from the SMTP server or an issue with the alert rule in ADF by creating custom alert emails via logic app through ADF pipeline. 2. To send an aler...