Azure Data Factory

Transforming and Transferring Data at the Speed of Business

Need to update large quantities of records? ADF may be the Microsoft solution you’ve been looking for…

You have plenty of options when planning to “upsert” (insert and update) data into Dynamics 365, Power Platform, and the underlying Microsoft Dataverse structures. In addition to Dynamics 365’s own web-based application programming interface (API), there’s always its built-in data import feature and Power Platform’s dataflows service. However, if you need to migrate a large amount of information—say hundreds of thousands or even millions of rows of data—Azure Data Factory (ADF) is worth your consideration.

Azure Data Factory (ADF) is a cloud-based web integration application designed for transferring and editing data at scale.

ADF is ideal for situations where you need to migrate or bulk update large quantities of data, especially if you’re already using other Azure services as it can handle larger volumes of data, offers improved performance, and can be less expensive than building for Dynamics 365 or Power App APIs.

Large Volumes of Data

While there are several applications you can use to connect, transfer, and bulk edit your data, not all of them have the capacity to manage massive volumes. These are the types of situations where ADF excels. Azure Data Factory makes it possible to load and export millions of rows of data into Microsoft Dynamics 365 and Dataverse.

Need for Speed
ADF includes a Dynamics 365 connector that supports multiple connections and record batching capabilities, allowing you to upsert data at a remarkably quick rate. We’ve seen it operate at speeds of up to 500K records per hour.

Getting Started As Soon As Possible 
While you can engage a technology consultant like Altriva to build a custom application that will transfer or transform your data, Azure Data Factory is a reasonably priced option that features pre-built connectors for dozens of databases. It’s a powerful option that can help get you up and running with a shorter development runway than a fully custom solution.

We find ADF’s Copy Activity, Error Reviewing, and Mass Migration features particularly worth taking advantage of…

Copy Activity
Azure Data Factory includes a feature called "Copy Activity" which enables users to copy data from one database to another at scale. Copy Activity works for more than just Microsoft products like Azure SQL server or Azure Storage; it can also copy data from an on-premises Oracle Database or flat files into Dynamics 365.

Additionally, we encourage you to take advantage of the Copy Activity status page, which shows the number of source rows that ADF retrieved so far, the number of rows upserted in Dynamics 365 / Dataverse and the total time the process has taken. This status page makes it easy to keep track of the migration process.

Error Reviewing
Another way that ADF keeps the user informed about the migration status is through its ability to write errors or warnings to Azure Storage. It allows the user to review row-level problems with the migration and take corrective action.

Long-Running Migration Capabilities
For significant data migration projects, such as those involving millions of rows of data, it is often crucial that the migration run 24/7 and uninterrupted. For example, this might be the case when a business is transitioning from an old system on a Friday and wants employees to start using Dynamics 365 the following Monday. ADF can not only automatically retry failed upsert operations to help keep it running consistently, but also utilize advanced capabilities, such as looping controls, advanced error handling and alerts to help a data migration team to meet an important deadline.

Here are a few common situations where you might want to consider using Azure Data Factory:

• Migrating information from one database to or from Dynamics 365 / Dataverse
• Copying data-only from a production environment into a sandbox environment
• Regularly scheduled updates, such as refreshing a system’s records every day at midnight
• Bulk editing records (exporting, transforming, and re-uploading information)

Although Azure Data Factory can most data migration requirements, you should be aware that it may not have every capability you want for copying data to D365 on its own.

For example, if your data migration involves copying support case records into D365, you will likely want the case records in Dynamics to be marked as closed or resolved. ADF does not provide an option to close case records in Dynamics 365, however, it does provide the ability to call Azure Functions to perform these types of operations on migrated data.

Interested in learning more about how Azure Data Factory could be used at your organization? Read the Microsoft documentation or get in touch with an Altriva consultant.

• The Latest Updates to Power Apps Portals
• How to Connect Excel to Dynamics 365 with Power Query
• Voice Channel for Dynamics 365 Customer Service

Altriva Solutions is an award-winning business technology consulting firm. Founded on CRM and Microsoft technologies, we help companies achieve their business objectives through smart, integrated, customer-centric solutions.