Author: Madhukar Gilla

Pagination in cosmos db

the following code snippet details on how to do pagination is cosmos db , we just need to pass a pagesize and the continuation token, when we are calling the method for the first time, we can just pass in a null and for the subsequent requests, we can pass in the continuation token, which is returned as a part of the response of this method.

 public async Task<ProductResponse> GetAllProducts(int pageSize, string continuationToken)
        {
            List<Product> products = new List<Product>();
            ProductsResponse productsResponse = new ProductsResponse();
            QueryDefinition queryDef = new QueryDefinition("select * from Prodcuts p");
            string token = null;
            if (!string.IsNullOrWhiteSpace(continuationToken))
                token = continuationToken;

            using FeedIterator<Product> resultSet = _container.GetItemQueryIterator<Product>(queryDefinition: queryDef, continuationToken: token, new QueryRequestOptions { MaxItemCount = pageSize });
            {
                var items = await resultSet.ReadNextAsync();
                foreach (Product item in items)
                {
                    products.Add(item);

                }
  productsResponse.ContinuationToken = items.ContinuationToken;
            }

            productResponse.Products = products;
            return productResponse;
        }

Azure Bicep template for Azure Cosmos DB serverless

Azure Cosmos DB serverless went GA last week, but the documentation has not caught up yet, if you are looking to provision an Azure Cosmos DB serverless account using Bicep you can use the following snippet, the magic here is the capabilities section. 

resource cosmos 'Microsoft.DocumentDB/databaseAccounts@2021-04-15' = {
  name: cosmosName
  location: location
  kind: 'GlobalDocumentDB'
  properties: {
    //enableFreeTier: true
    consistencyPolicy: {
      defaultConsistencyLevel: 'Session'
    }
    locations: [
      {
        locationName: location
        failoverPriority: 0
        isZoneRedundant: false
      }
    ]
    capabilities: [
      {
        name: 'EnableServerless'
      }
    ]
    databaseAccountOfferType: 'Standard'
  }
}

Handling azure data bricks errors in azure datafactory

issue: when an exception happens in azure datafactory notebooks only just a url contianing the details is sent back to the datafactory, it does not go to the error flow.

one of the solutions we implemented was to catch an exception in the notebook is to send a json string containing the following

{

“isError”:”true”

“ExceptionDetails”:”some exception happend”

}

then datafactory will parse the json object log it if it wants and then choose the appropriate flow to handle the exception.

if you want to terminate the pipeline then do something like the following, where in use dynamic sql to raise an error.

azure data factory 2 – How do I raise an error from if activity in ADFV2? – Stack Overflow

hope this helps..

Http wrapper for Terraform CLI Part 1

OOB Hashicorp doesn’t offer an Terraform HTTP API/SDK for provisioning resources to cloud providers using terrafrom templates,(Their proprietary cloud does an awesome job of doing this).

However on the other hand Azure and AWS have an API (SDK) for provisioning resources using ARM and cloud formation templates.

For this POC i went ahead and created a wrapper for the terraform CLI to create an resource group in Azure.

The wrapper is written in Azure C# functions running locally, the second post of this series will cover the same functions running in Azure (hopefully).

The steps that i took are as follows

  1. Installed AZURE CLI and logged in.
  2. Created a Azure Functions Project
  3. Copied the terraform template, and the terraform cli into windows temp folder (C:\Users\<<username>>\AppData\Local\Temp)
  4. And the code is below and self explanatory, used the StartProcess method (and i did something similar Invoke Btstask from .net code – madhukar gilla (wordpress.com) in Dec 2010)
terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "~>2.0"
    }
  }
}
provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "myterraformgroup" {
    name     = "myResourceGroup"
    location = "eastus"

    tags = {
        environment = "Terraform Demo"
    }
}
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

namespace TerraformHttpWrapper
{
    public static class Function2
    {
        [FunctionName("Function2")]
        public static async Task<List<string>> RunOrchestrator(
            [OrchestrationTrigger] IDurableOrchestrationContext context)
        {
            var outputs = new List<string>();

            // Replace "hello" with the name of your Durable Activity Function.
            outputs.Add(await context.CallActivityAsync<string>("Function2_Hello", "Tokyo"));
            //outputs.Add(await context.CallActivityAsync<string>("Function2_Hello", "Seattle"));
            //outputs.Add(await context.CallActivityAsync<string>("Function2_Hello", "London"));

            // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
            return outputs;
        }

        [FunctionName("Function2_Hello")]
        public static string SayHello([ActivityTrigger] string name, ILogger log, ExecutionContext context)
        {
            log.LogInformation($"Saying hello to {name}.");

            log.LogInformation("C# HTTP trigger function processed a request.");


            var tempRootDir = Path.GetTempPath();


            Directory.SetCurrentDirectory(tempRootDir);

            log.LogInformation($"----------------------------- Current directory { Directory.GetCurrentDirectory()} --------------------");
            try
            {
                Process p = new Process();
                p.StartInfo.UseShellExecute = false;
                p.StartInfo.FileName = "terraform.exe";
                p.StartInfo.Arguments = "init";
                p.StartInfo.RedirectStandardOutput = true;

                log.LogInformation("*****************************" + Environment.CurrentDirectory);

                p.Start();
                log.LogInformation(p.StandardOutput.ReadToEnd());
                p.WaitForExit();


                log.LogInformation("----------------------------- DONE init --------------------");

                Process ptwo = new Process();
                ptwo.StartInfo.UseShellExecute = false;
                ptwo.StartInfo.FileName = "terraform.exe";
                ptwo.StartInfo.Arguments = "plan  -detailed-exitcode -lock=false";
                ptwo.StartInfo.RedirectStandardOutput = true;

                ptwo.Start();
                log.LogInformation(ptwo.StandardOutput.ReadToEnd());
                ptwo.WaitForExit();

                log.LogInformation("----------------------------- DONE plan --------------------");

                Process pthree = new Process();
                pthree.StartInfo.UseShellExecute = false;
                pthree.StartInfo.FileName = "terraform.exe";
                pthree.StartInfo.Arguments = "apply -auto-approve -lock=false";
                pthree.StartInfo.RedirectStandardOutput = true;

                pthree.Start();
                log.LogInformation(pthree.StandardOutput.ReadToEnd());
                pthree.WaitForExit();

                //log.LogInformation(output);

                log.LogInformation("----------------------------- DONE apply --------------------");

            }
            catch (Exception ed)
            {
                log.LogInformation("Error: ." + ed);
            }

            return name;
        }

        [FunctionName("Function2_HttpStart")]
        public static async Task<HttpResponseMessage> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestMessage req,
            [DurableClient] IDurableOrchestrationClient starter,
            ILogger log)
        {
            // Function input comes from the request content.
            string instanceId = await starter.StartNewAsync("Function2", null);

            log.LogInformation($"Started orchestration with ID = '{instanceId}'.");

            return starter.CreateCheckStatusResponse(req, instanceId);
        }
    }
}

And the logs from the Azure Function and terraform cli are as follows.

Terrafrom and Azure Functions logs

And the created function group

ffmpeg, my favourite tool for audio editing.

As you know based on my previous posts that i recently bought a fitbit ionic for inbuilt GPS and music player, i have been using playerfm on mobile forever, one of the nice features of palyerfm is to increase/decrease the audio speed, and i am habituated to listening at 1.2x speed, now without these option in the native ionic music player, i had to fallback on editing the podcasts on my machine by using ffmpeg.

All you have to do is run the following cmd, use the atempo parameter to adjust the speed accordingly.

ffmpeg -i input.mp3 -filter:a “atempo=1.2” -vn output.mp3

its all worth it because i don’t have to carry my mobile phone for those long walks or jogs !!

Fitbit Ionic initial setup and firmware update…

Recently bought an Fitbit ionic, reason being

  1. I don’t need to carry a phone or another device for music and GPS.

the initial setup was a bummer thought of returning it back, the only way I could update the firmware was doing the following

  1.  Have 3g on the phone.
  2. Make the phone a hotspot
  3. Install the Fitbit app on Mac or windows(blue tooth).
  4. Add a new device and run the setup through the computer.
  5. Configure Fitbit WiFi to use to the Mobile Hotspot from step 2.
  6. And on the update configuration screen just select activity tracking, leave the others they will get automatically installed.

That’s it.. well then what combination did not work.

  1. With the app on windows configuring the Fitbit WiFi to use  Hotel/Home/Office WiFi did not work all were 2.4 GHz.
  2. Tried the same setup above with a phone instead of an windows box did not work.
  3. Tried both the above steps with Bluetooth instead of WiFi did not work.


Alll the best and post comments for any other information or questions.

Recruitment

I have been interviewing for a while, for my team for other teams…etc, most of the applicants are more than apt with their technical skills and logical reasoning, if you are still not sure about their technical skill set ask them to do a quick hands on exercise or ask them give a code walkthrough of their github project.

But how do we decide on who to hire? what we should be asking ourselves is “would I have him on my team” if you answer yes then nothing else matters just go ahead and select him. Technical skill can be worked upon if the applicant is a good fit for you team and yes apart from their skill set your gut feeling does matter.

Azure API Management Versions Vs Revisions

With the new Version and Revision features its confusing on when to use revisions or versions, my take is as follows

When should you use  Revisions:

  • If you want to do changes that are above your backend services, for example using policies.
    • Add authentication: your api did not have any authentication and you want to have one without disturbing your existing service, so you would add that in the in the new revision
    • Logging: you api did not have any logging of incoming and outgoing requests and you don’t want to disturb your backend service so you create a revision for that feature
  • Note: only 1 revision is exposed to the outside world

When should you use Versions:

  • Use versions when you introduce a breaking change in your api and deploy it as a new backend server and want to support both those deployments.