{"id":973,"date":"2021-05-13T13:43:13","date_gmt":"2021-05-13T11:43:13","guid":{"rendered":"https:\/\/www.flip-design.de\/?p=973"},"modified":"2021-05-14T08:53:38","modified_gmt":"2021-05-14T06:53:38","slug":"stream-analytics-and-power-bi","status":"publish","type":"post","link":"https:\/\/www.flip-design.de\/?p=973","title":{"rendered":"Stream Analytics and Power BI"},"content":{"rendered":"\n<p>Azure Stream Analytics can used to stream data to different targets and to Power BI. In this post, I will show how to stream data to a Power BI dataset from a Blob storage. Normally you will get at this scenario data from IoT devices or something which push very frequently data, but to make this post a little bit easier for me for you, when you want to reproduce something of this, it can be created with resources which are normally available. Okay, let us start to build up the environment.<\/p>\n\n\n\n<p>First, I create a storage account. Please make sure, that you create this account in an area, where you also want to create the Stream Analytics. This is important to make the setup easier and to save money and minimize the latency.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"382\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image.png\" alt=\"\" class=\"wp-image-974\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-300x189.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-475x300.png 475w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>Next, we need our data intake container where we store the data which will imported into out Power BI environment.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-1.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"203\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-1.png\" alt=\"\" class=\"wp-image-975\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-1.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-1-300x101.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-1-500x168.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>To this container I upload a simple CSV file<br><code>Category;Date;OrderId;Amount<br>Trucks;9\/5\/2021;763;2289<\/code><br>Imagine that your store will send you for ever sale a file which should be immediately available in your Power BI report.<\/p>\n\n\n\n<p>Next, we need our Stream Analytics Job, you will find this resource at the Azure marketplace:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-2.png\"><img decoding=\"async\" loading=\"lazy\" width=\"377\" height=\"150\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-2.png\" alt=\"\" class=\"wp-image-976\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-2.png 377w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-2-300x119.png 300w\" sizes=\"(max-width: 377px) 100vw, 377px\" \/><\/a><\/figure>\n\n\n\n<p>Please ensure that you create this job at the same region as you created the storage account. By creating this job, I leave everything by default. The Units are by default at 3 and can scaled up by need and after creating this resource.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-3.png\"><img decoding=\"async\" loading=\"lazy\" width=\"515\" height=\"527\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-3.png\" alt=\"\" class=\"wp-image-977\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-3.png 515w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-3-293x300.png 293w\" sizes=\"(max-width: 515px) 100vw, 515px\" \/><\/a><\/figure>\n\n\n\n<p>After a successful deployment we need to configure the data in-take. You can configure this by navigating zo input and select \u201cnew\u201d and Blob storage. For the Authentication mode I select \u201cconnection string\u201d. By selecting this, maybe you need to provide a key, this key can be grab from your storage account under access keys. Please make also sure, that you configure the data format to CSV with semicolon as a delimiter.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-4.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-4.png\" alt=\"\" class=\"wp-image-978\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-4.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-4-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-4-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-5.png\"><img decoding=\"async\" loading=\"lazy\" width=\"381\" height=\"75\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-5.png\" alt=\"\" class=\"wp-image-979\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-5.png 381w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-5-300x59.png 300w\" sizes=\"(max-width: 381px) 100vw, 381px\" \/><\/a><\/figure>\n\n\n\n<p>Now we need our output. For this I created at my Power BI tenant only a new workspace. You can use an existing, but to get a better overview for myself, I created a new one. By selecting outputs, I created a new with Power BI.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-6.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-6.png\" alt=\"\" class=\"wp-image-980\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-6.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-6-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-6-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>By going back to the Overview, we can edit out query which selecting our data from the input and push the data to the output.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-7.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"330\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-7.png\" alt=\"\" class=\"wp-image-981\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-7.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-7-300x164.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-7-500x273.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>Now everything is ready, and we can start the job. After that, and we provide new data, we get a new dataset in our Power Bi workspace where we get our data from the blob storage. To provide new data, you can easily upload a new file. You can check this at the overview page.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-8.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"249\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-8.png\" alt=\"\" class=\"wp-image-982\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-8.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-8-300x123.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-8-500x206.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p>By creating a new report at Power BI based on the new dataset, you can check the data. The dataset contains additionally some information about the data file and the import process. I think it makes more fun when you automatically get new data, for this approach I created a PowerShell script. You need by configuring this script also the access key from your blob storage. After that, you can run this script in a loop.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$accesskey = \"f1vxAsiHHWvdEF8lfAExLq66rghWfqzXZoR3ugq9p5qnqDTnIt4DAUvz4luHEJ5w1Gbcg+yy6jT+EEvHJGaXtA==\"\n$StorageAccountName = \"streamingdata0105\"\n$containerName = \"data\"\n\n$date = Get-Date\n$category = \"Trucks\"\n$amount = $date.Millisecond\n\n\n$filename = $category + $date.Hour + $date.Minute + $date.Second + \".csv\"\n$path = \"c:\\temp\\files\\\"\n$tempfilepath = \"c:\\temp\\files\\\" + $filename\n\n$output =  @()\n$output =  @(\"Category;Date;OrderId;Amount\")\n$output += @($category + \";\" + $date.Day + \"\/\" + $date.Month + \"\/\" + $date.Year + \";\" + $amount + \";\" + $amount * 3) \n$output | Out-File -FilePath $tempfilepath\n\nConnect-AzAccount\n$StorageContext = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $accesskey \nSet-AzStorageBlobContent -File $tempfilepath -Container $containerName -Blob $filename -Context $StorageContext <\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-9.png\"><img decoding=\"async\" loading=\"lazy\" width=\"605\" height=\"284\" src=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-9.png\" alt=\"\" class=\"wp-image-983\" srcset=\"https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-9.png 605w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-9-300x141.png 300w, https:\/\/www.flip-design.de\/wp-content\/uploads\/2021\/05\/image-9-500x235.png 500w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/figure>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Azure Stream Analytics can used to stream data to different targets and to Power BI. In this post, I will show how to stream data to a Power BI dataset from a Blob storage. Normally you will get at this &hellip; <a href=\"https:\/\/www.flip-design.de\/?p=973\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts\/973"}],"collection":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=973"}],"version-history":[{"count":2,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts\/973\/revisions"}],"predecessor-version":[{"id":985,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=\/wp\/v2\/posts\/973\/revisions\/985"}],"wp:attachment":[{"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=973"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=973"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flip-design.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=973"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}