Making of insta-photo: Image analysis with the Computer Vision API

I wanted to try the Microsoft Cognitive Services since quite some time and I finally found some free time on a weekend a few weeks ago.

insta-photo

insta-photo analyzes an image with the Computer Vision API and displays the caption and tags which can be posted to social media, e.g. Instagram.

Technology stack

• Angular
• Angular Material
• Cognitive Services
• Computer Vision API: analyze, describe
• TypeScript model generated with json2ts

The source code can be found here on Github.

I created a new project with Angular, added Angular Material and generated a new vision component for the user interface. I generated a service that handles the Vision API call request and response.

The vision API url and key are stored in the environment file which should not be shown on the client side in production because anyone can see the key and use the service with this key. The url is used in the http post and the key is sent in the http options.

The user can select an image and a preview is displayed in an img html element using the FileReader.readAsDataUrl. The user can upload it to the Computer Vision API for analysis. The result is formatted into a caption that displays the text and the tags that the Vision API returns.

Getting a valid response from the Computer Vision API has been challenging and it took me some time to figure it out. I got different error codes, and in particular Error code 400 invalid image. I used Chrome Developer Tools and Fiddler to investigate the response error. Additionally large images are not supported by the Vision API.

Seamless integration with Instagram in this web page is not possible because Instagram does not allow automatic posting and the Instagram API does not include this possibility. Instead images can only be posted manually. However Instagram for Business supports this feature but it is not for free.