{"id":1057,"date":"2021-09-19T13:06:05","date_gmt":"2021-09-19T11:06:05","guid":{"rendered":"https:\/\/www.flip-design.de\/?p=1057"},"modified":"2021-09-19T13:06:05","modified_gmt":"2021-09-19T11:06:05","slug":"trigger-an-azure-data-factory-pipeline-by-external-users","status":"publish","type":"post","link":"https:\/\/www.flip-design.de\/?p=1057","title":{"rendered":"Trigger an Azure Data Factory pipeline by external users"},"content":{"rendered":"\n<p>The very simple and common use case is, that you have a pipeline that fills your data warehouse, process your cube, and delivers this information to your business users. The data warehouse uses as a part of it, excel files which are created by the business. The pipeline runs every day one or two times, but now, there is a requirement, that the process owner of these excel files what to control this process by trigger the pipeline. The reason is very simple, there was a file which contains errors and must be replaced that the management does not get misleading information.<br>If you want that the user can trigger a ADF pipeline you have different options, you can give him permissions or you can trigger the ADF by an event. I think, the last one is more interesting, because the user needs no permissions to the factory.<\/p>\n\n\n\n<p>So, I have created an Azure storage account. This account is only to save a trigger file which is created by the user in this container. Next, I have created a simple pipeline which needs to be triggered by the business<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-5.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-5.png\" alt=\"\" class=\"wp-image-1058\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-5.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-5-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-5-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>The pipeline deletes after the run the trigger file from the storage. I use an Azure Event Blob trigger with these settings<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-6.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-6.png\" alt=\"\" class=\"wp-image-1059\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-6.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-6-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-6-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>So, I a place a file which starts with the word \u201ctrigger\u201d inside the filename at the container, the ADF will recognize it at runs the pipeline.<br>After uploading a file named \u201ctriggeron.txt\u201d the pipeline will run immediately<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-7.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-7.png\" alt=\"\" class=\"wp-image-1060\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-7.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-7-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-7-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-8.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-8.png\" alt=\"\" class=\"wp-image-1061\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-8.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-8-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-8-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>Now, you need only an option that the user can create this file very easily. There you can provide a Power Shell script to this user with access key, or you can use a Power Automate flow. I think this is better, because you have a flow which is shared to this user and when it runs, it will create this file inside this container. You can also trigger this flow by arriving an email to a particular post box, like this flow<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-9.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-9.png\" alt=\"\" class=\"wp-image-1062\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-9.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-9-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/09\/image-9-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>But you must aware, that in the case you are using Power Automate you need a Premium connector.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The very simple and common use case is, that you have a pipeline that fills your data warehouse, process your cube, and delivers this information to your business users. The data warehouse uses as a part of it, excel files &hellip; <a href=\"https:\/\/www.flip-design.de\/?p=1057\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts\/1057"}],"collection":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1057"}],"version-history":[{"count":1,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts\/1057\/revisions"}],"predecessor-version":[{"id":1063,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts\/1057\/revisions\/1063"}],"wp:attachment":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}