Copy data from Xero using Azure Data Factory or Synapse Analytics

Article
07/12/2024

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

This article outlines how to use the Copy Activity in an Azure Data Factory or Synapse Analytics pipeline to copy data from Xero. It builds on the copy activity overview article that presents a general overview of copy activity.

Note

The Xero connector requires OAuth authentication and is not intended for server-to-server use.

Supported capabilities

This Xero connector is supported for the following capabilities:

Supported capabilities	IR
Copy activity (source/-)	① ②
Lookup activity	① ②

① Azure integration runtime ② Self-hosted integration runtime

For a list of data stores that are supported as sources/sinks, see the Supported data stores table.

Specifically, this Xero connector supports:

OAuth 2.0 and OAuth 1.0 authentication. For OAuth 1.0, the connector supports Xero private application but not public application.
All Xero tables (API endpoints) except "Reports".

Getting started

To perform the Copy activity with a pipeline, you can use one of the following tools or SDKs:

Create a linked service to Xero using UI

Use the following steps to create a linked service to Xero in the Azure portal UI.

Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then click New:
- Azure Data Factory
- Azure Synapse
Search for Xero and select the Xero connector.
Configure the service details, test the connection, and create the new linked service.

Connector configuration details

The following sections provide details about properties that are used to define Data Factory entities specific to Xero connector.

Linked service properties

The following properties are supported for Xero linked service:

Property	Description	Required
type	The type property must be set to: Xero	Yes
connectionProperties	A group of properties that defines how to connect to Xero.	Yes
Under `connectionProperties`:
host	The endpoint of the Xero server (`api.xero.com`).	Yes
authenticationType	Allowed values are `OAuth_2.0` and `OAuth_1.0`.	Yes
consumerKey	For OAuth 2.0, specify the client ID for your Xero application. For OAuth 1.0, specify the consumer key associated with the Xero application. Mark this field as a SecureString to store it securely, or reference a secret stored in Azure Key Vault.	Yes
privateKey	For OAuth 2.0, specify the client secret for your Xero application. For OAuth 1.0, specify the private key from the .pem file that was generated for your Xero private application. Note to generate the privatekey.pem with numbits of 512 using `openssl genrsa -out privatekey.pem 512`, 1024 is not supported. Include all the text from the .pem file including the Unix line endings(\n), see sample below. Mark this field as a SecureString to store it securely, or reference a secret stored in Azure Key Vault.	Yes
tenantId	The tenant ID associated with your Xero application. Applicable for OAuth 2.0 authentication. Learn how to get the tenant ID from Check the tenants you're authorized to access section.	Yes for OAuth 2.0 authentication
refreshToken	Applicable for OAuth 2.0 authentication. The OAuth 2.0 refresh token is associated with the Xero application and used to refresh the access token; the access token expires after 30 minutes. Learn about how the Xero authorization flow works and how to get the refresh token from this article. To get a refresh token, you must request the offline_access scope. Know limitation: Note Xero resets the refresh token after it's used for access token refresh. For operationalized workload, before each copy activity run, you need to set a valid refresh token for the service to use. Mark this field as a SecureString to store it securely, or reference a secret stored in Azure Key Vault.	Yes for OAuth 2.0 authentication
useEncryptedEndpoints	Specifies whether the data source endpoints are encrypted using HTTPS. The default value is true.	No
useHostVerification	Specifies whether the host name is required in the server's certificate to match the host name of the server when connecting over TLS. The default value is true.	No
usePeerVerification	Specifies whether to verify the identity of the server when connecting over TLS. The default value is true.	No

Example: OAuth 2.0 authentication

{
    "name": "XeroLinkedService",
    "properties": {
        "type": "Xero",
        "typeProperties": {
            "connectionProperties": { 
                "host": "api.xero.com",
                "authenticationType":"OAuth_2.0", 
                "consumerKey": {
                    "type": "SecureString",
                    "value": "<client ID>"
                },
                "privateKey": {
                    "type": "SecureString",
                    "value": "<client secret>"
                },
                "tenantId": "<tenant ID>", 
                "refreshToken": {
                    "type": "SecureString",
                    "value": "<refresh token>"
                }, 
                "useEncryptedEndpoints": true, 
                "useHostVerification": true, 
                "usePeerVerification": true
            }            
        }
    }
}

Example: OAuth 1.0 authentication

{
    "name": "XeroLinkedService",
    "properties": {
        "type": "Xero",
        "typeProperties": {
            "connectionProperties": {
                "host": "api.xero.com", 
                "authenticationType":"OAuth_1.0", 
                "consumerKey": {
                    "type": "SecureString",
                    "value": "<consumer key>"
                },
                "privateKey": {
                    "type": "SecureString",
                    "value": "<private key>"
                }, 
                "useEncryptedEndpoints": true,
                "useHostVerification": true,
                "usePeerVerification": true
            }
        }
    }
}

Sample private key value:

Include all the text from the .pem file including the Unix line endings(\n).

"-----BEGIN RSA PRIVATE KEY-----\nMII***************************************************P\nbu****************************************************s\nU/****************************************************B\nA*****************************************************W\njH****************************************************e\nsx*****************************************************l\nq******************************************************X\nh*****************************************************i\nd*****************************************************s\nA*****************************************************dsfb\nN*****************************************************M\np*****************************************************Ly\nK*****************************************************Y=\n-----END RSA PRIVATE KEY-----"

Dataset properties

For a full list of sections and properties available for defining datasets, see the datasets article. This section provides a list of properties supported by Xero dataset.

To copy data from Xero, set the type property of the dataset to XeroObject. The following properties are supported:

Property	Description	Required
type	The type property of the dataset must be set to: XeroObject	Yes
tableName	Name of the table.	No (if "query" in activity source is specified)

Example

{
    "name": "XeroDataset",
    "properties": {
        "type": "XeroObject",
        "typeProperties": {},
        "schema": [],
        "linkedServiceName": {
            "referenceName": "<Xero linked service name>",
            "type": "LinkedServiceReference"
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by Xero source.

Xero as source

To copy data from Xero, set the source type in the copy activity to XeroSource. The following properties are supported in the copy activity source section:

Property	Description	Required
type	The type property of the copy activity source must be set to: XeroSource	Yes
query	Use the custom SQL query to read data. For example: `"SELECT * FROM Contacts"`.	No (if "tableName" in dataset is specified)

Example:

"activities":[
    {
        "name": "CopyFromXero",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<Xero input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "XeroSource",
                "query": "SELECT * FROM Contacts"
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

Note the following when specifying the Xero query:

Tables with complex items will be split to multiple tables. For example, Bank transactions has a complex data structure "LineItems", so data of bank transaction is mapped to table Bank_Transaction and Bank_Transaction_Line_Items, with Bank_Transaction_ID as foreign key to link them together.
Xero data is available through two schemas: Minimal (default) and Complete. The Complete schema contains prerequisite call tables which require additional data (e.g. ID column) before making the desired query.

The following tables have the same information in the Minimal and Complete schema. To reduce the number of API calls, use Minimal schema (default).

Bank_Transactions
Contact_Groups
Contacts
Contacts_Sales_Tracking_Categories
Contacts_Phones
Contacts_Addresses
Contacts_Purchases_Tracking_Categories
Credit_Notes
Credit_Notes_Allocations
Expense_Claims
Expense_Claim_Validation_Errors
Invoices
Invoices_Credit_Notes
Invoices_ Prepayments
Invoices_Overpayments
Manual_Journals
Overpayments
Overpayments_Allocations
Prepayments
Prepayments_Allocations
Receipts
Receipt_Validation_Errors
Tracking_Categories

The following tables can only be queried with complete schema:

Complete.Bank_Transaction_Line_Items
Complete.Bank_Transaction_Line_Item_Tracking
Complete.Contact_Group_Contacts
Complete.Contacts_Contact_ Persons
Complete.Credit_Note_Line_Items
Complete.Credit_Notes_Line_Items_Tracking
Complete.Expense_Claim_ Payments
Complete.Expense_Claim_Receipts
Complete.Invoice_Line_Items
Complete.Invoices_Line_Items_Tracking
Complete.Manual_Journal_Lines
Complete.Manual_Journal_Line_Tracking
Complete.Overpayment_Line_Items
Complete.Overpayment_Line_Items_Tracking
Complete.Prepayment_Line_Items
Complete.Prepayment_Line_Item_Tracking
Complete.Receipt_Line_Items
Complete.Receipt_Line_Item_Tracking
Complete.Tracking_Category_Options

Lookup activity properties

To learn details about the properties, check Lookup activity.

For a list of supported data stores by the copy activity, see supported data stores.