Python web and worker roles with Python Tools for Visual Studio

This article provides an overview of using Python web and worker roles using Python Tools for Visual Studio. Learn how to use Visual Studio to create and deploy a basic Cloud Service that uses Python.

Prerequisites

• Visual Studio 2013, 2015, or 2017
• Python Tools for Visual Studio (PTVS)
• Azure SDK Tools for Visual Studio (VS) 2013 or
  Azure SDK Tools for VS 2015 or
  Azure SDK Tools for VS 2017
• Python 2.7 32-bit or Python 3.8 32-bit

What are Python web and worker roles?

Azure provides three compute models for running applications: Web Apps feature in Azure App Service, Azure Virtual Machines, and Azure Cloud Services. All three models support Python. Cloud Services, which include web and worker roles, provide Platform as a Service (PaaS). Within a cloud service, a web role provides a dedicated Internet Information Services (IIS) web server to host front end web applications. A worker role can run asynchronous, long-running, or perpetual tasks independent of user interaction or input.

For more information, see What is a Cloud Service?.

Project creation

In Visual Studio, you can select Azure Cloud Service in the New Project dialog box, under Python.

In the Azure Cloud Service wizard, you can create new web and worker roles.

The worker role template comes with boilerplate code to connect to an Azure storage account or Azure Service Bus.

You can add web or worker roles to an existing cloud service at any time. You can choose to add existing projects in your solution, or create new ones.

Your cloud service can contain roles implemented in different languages. For example, you can have a Python web role implemented using Django, with Python, or with C# worker roles. You can easily communicate between your roles using Service Bus queues or storage queues.

Install Python on the cloud service

The main problem with the setup scripts is that they don't install Python. First, define two startup tasks in the ServiceDefinition.csdef file. The first task (PrepPython.ps1) downloads and installs the Python runtime. The second task (PipInstaller.ps1) runs pip to install any dependencies you may have.

The following scripts were written targeting Python 3.8. If you want to use the version 2.x of python, set the PYTHON2 variable file to on for the two startup tasks and the runtime task: <Variable name="PYTHON2" value="<mark>on</mark>" />.

<Startup>

  <Task executionContext="elevated" taskType="simple" commandLine="bin\ps.cmd PrepPython.ps1">
    <Environment>
      <Variable name="EMULATED">
        <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
      </Variable>
      <Variable name="PYTHON2" value="off" />
    </Environment>
  </Task>

  <Task executionContext="elevated" taskType="simple" commandLine="bin\ps.cmd PipInstaller.ps1">
    <Environment>
      <Variable name="EMULATED">
        <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
      </Variable>
      <Variable name="PYTHON2" value="off" />
    </Environment>

  </Task>

</Startup>

The PYTHON2 and PYPATH variables must be added to the worker startup task. The PYPATH variable is only used if the PYTHON2 variable is set to on.

<Runtime>
  <Environment>
    <Variable name="EMULATED">
      <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
    </Variable>
    <Variable name="PYTHON2" value="off" />
    <Variable name="PYPATH" value="%SystemDrive%\Python27" />
  </Environment>
  <EntryPoint>
    <ProgramEntryPoint commandLine="bin\ps.cmd LaunchWorker.ps1" setReadyOnProcessStart="true" />
  </EntryPoint>
</Runtime>

Sample ServiceDefinition.csdef

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="AzureCloudServicePython" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2015-04.2.6">
  <WorkerRole name="WorkerRole1" vmsize="Small">
    <ConfigurationSettings>
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" />
      <Setting name="Python2" />
    </ConfigurationSettings>
    <Startup>
      <Task executionContext="elevated" taskType="simple" commandLine="bin\ps.cmd PrepPython.ps1">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="PYTHON2" value="off" />
        </Environment>
      </Task>
      <Task executionContext="elevated" taskType="simple" commandLine="bin\ps.cmd PipInstaller.ps1">
        <Environment>
          <Variable name="EMULATED">
            <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
          </Variable>
          <Variable name="PYTHON2" value="off" />
        </Environment>
      </Task>
    </Startup>
    <Runtime>
      <Environment>
        <Variable name="EMULATED">
          <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
        </Variable>
        <Variable name="PYTHON2" value="off" />
        <Variable name="PYPATH" value="%SystemDrive%\Python27" />
      </Environment>
      <EntryPoint>
        <ProgramEntryPoint commandLine="bin\ps.cmd LaunchWorker.ps1" setReadyOnProcessStart="true" />
      </EntryPoint>
    </Runtime>
    <Imports>
      <Import moduleName="RemoteAccess" />
      <Import moduleName="RemoteForwarder" />
    </Imports>
  </WorkerRole>
</ServiceDefinition>

Next, create the PrepPython.ps1 and PipInstaller.ps1 files in the ./bin folder of your role.

PrepPython.ps1

This script installs python. If the PYTHON2 environment variable is set to on, then Python 2.7 is installed, otherwise Python 3.8 is installed.

[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls"
$is_emulated = $env:EMULATED -eq "true"
$is_python2 = $env:PYTHON2 -eq "on"
$nl = [Environment]::NewLine

if (-not $is_emulated){
    Write-Output "Checking if Python is installed...$nl"
    if ($is_python2) {
        & "${env:SystemDrive}\Python27\python.exe"  -V | Out-Null
    }
    else {
        py -V | Out-Null
    }

    if (-not $?) {

        $url = "https://www.python.org/ftp/python/3.8.8/python-3.8.8-amd64.exe"
        $outFile = "${env:TEMP}\python-3.8.8-amd64.exe"

        if ($is_python2) {
            $url = "https://www.python.org/ftp/python/2.7.18/python-2.7.18.amd64.msi"
            $outFile = "${env:TEMP}\python-2.7.18.amd64.msi"
        }

        Write-Output "Not found, downloading $url to $outFile$nl"
        Invoke-WebRequest $url -OutFile $outFile
        Write-Output "Installing$nl"

        if ($is_python2) {
            Start-Process msiexec.exe -ArgumentList "/q", "/i", "$outFile", "ALLUSERS=1" -Wait
        }
        else {
            Start-Process "$outFile" -ArgumentList "/quiet", "InstallAllUsers=1" -Wait
        }

        Write-Output "Done$nl"
    }
    else {
        Write-Output "Already installed"
    }
}

PipInstaller.ps1

This script calls up pip and installs all of the dependencies in the requirements.txt file. If the PYTHON2 environment variable is set to on, then Python 2.7 is used, otherwise Python 3.8 is used.

$is_emulated = $env:EMULATED -eq "true"
$is_python2 = $env:PYTHON2 -eq "on"
$nl = [Environment]::NewLine

if (-not $is_emulated){
    Write-Output "Checking if requirements.txt exists$nl"
    if (Test-Path ..\requirements.txt) {
        Write-Output "Found. Processing pip$nl"

        if ($is_python2) {
            & "${env:SystemDrive}\Python27\python.exe" -m pip install -r ..\requirements.txt
        }
        else {
            py -m pip install -r ..\requirements.txt
        }

        Write-Output "Done$nl"
    }
    else {
        Write-Output "Not found$nl"
    }
}

Modify LaunchWorker.ps1

The bin\LaunchWorker.ps1 was originally created to do a lot of prep work, but it doesn't really work. Replace the contents in that file with the following script.

This script calls the worker.py file from your Python project. If the PYTHON2 environment variable is set to on, then Python 2.7 is used, otherwise Python 3.8 is used.

$is_emulated = $env:EMULATED -eq "true"
$is_python2 = $env:PYTHON2 -eq "on"
$nl = [Environment]::NewLine

if (-not $is_emulated)
{
    Write-Output "Running worker.py$nl"

    if ($is_python2) {
        cd..
        iex "$env:PYPATH\python.exe worker.py"
    }
    else {
        cd..
        iex "py worker.py"
    }
}
else
{
    Write-Output "Running (EMULATED) worker.py$nl"

    # Customize to your local dev environment

    if ($is_python2) {
        cd..
        iex "$env:PYPATH\python.exe worker.py"
    }
    else {
        cd..
        iex "py worker.py"
    }
}

ps.cmd

The Visual Studio templates probably created a ps.cmd file in the ./bin folder. This shell script calls out the preceding PowerShell wrapper scripts and provides logging based on the name of the PowerShell wrapper called. If this file wasn't created, the following script would be in it:

@echo off

cd /D %~dp0

if not exist "%DiagnosticStore%\LogFiles" mkdir "%DiagnosticStore%\LogFiles"
%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Unrestricted -File %* >> "%DiagnosticStore%\LogFiles\%~n1.txt" 2>> "%DiagnosticStore%\LogFiles\%~n1.err.txt"

Run locally

If you set your cloud service project as the startup project and press F5, the cloud service runs in the local Azure emulator.

Although PTVS supports launching in the emulator, debugging (for example, breakpoints) doesn't work.

To debug your web and worker roles, you can set the role project as the startup project and debug that instead. You can also set multiple startup projects. Right-click the solution and then select Set StartUp Projects.

Publish to Azure

To publish, right-click the cloud service project in the solution and then select Publish.

Follow the wizard. If you need to, enable remote desktop. Remote desktop is helpful when you need to debug something.

When you finish configuring settings, choose Publish.

Some progress appears in the output window, then you see the Azure Activity Log window.

Deployment takes several minutes to complete, then your web and/or worker roles run on Azure!

Investigate logs

After the cloud service virtual machine starts up and installs Python, you can look at the logs to find any failure messages. These logs are located in the C:\Resources\Directory\{role}\LogFiles folder. PrepPython.err.txt has at least one error in it from when the script tries to detect if Python is installed and PipInstaller.err.txt may complain about an outdated version of pip.

Next steps

For more detailed information about working with web and worker roles in Python Tools for Visual Studio, see the PTVS documentation:

• Cloud Service Projects

For more details about using Azure services from your web and worker roles, such as using Azure Storage or Service Bus, see the following articles:

• Blob Service
• Queue Service
• Service Bus Queues
• Service Bus Topics