Skip to content

Lab050.md

Latest commit

 

History

History
61 lines (46 loc) · 5.19 KB

Lab050.md

File metadata and controls

61 lines (46 loc) · 5.19 KB
Raw

Integration Best Practices

The Integration features is basically Azure Data Factory (ADF), so whatever applies to ADF also applies to Synapse workspaces.

This is browser-based development

  • Restart your browser often
    • Occassionally a complex pipeline won't work correctly, even after you know you've fixed it. Simply save and commit your work, close your browser and start over. The DOM seems to get corrupted.
  • Use a separate browser for ADF development
    • If you use chrome as your daily-driver, consider using chromium, google chrome canary, firefox, edge, etc, for ADF work. You don't want to have to restart your browser due to ADF issues and lose your email windows, etc. Likewise, I know it shouldn't happen, but it seems like ADF sees more issues when I share the browser with other non-ADF activities.

Version Control

  • Never work without git. Never.
  • Consider using gitflow.

Should you have one one repo for:

  • each factory
  • all ADF factories
  • shared repo with other data projects ???

It really doesn't matter and is more a function of your development culture and release lifecycle. Some teams have monolith repos, some have individual items in their own repos. Since you generally have one DevOps pipeline per repo, you will have to deploy all items within that repo unless you segregate by folder structure. This would seem to indicate that the best practice is one repo/factory.

Team-based ADF development

  • How many data factories does a team of 3 need?
    • Answer: at least ONE data factory per developer
    • Why?: ADF is browser-based development and deploying a pipeline to an ADF instance while another developer is doing active development is a recipe for failure.
    • data factories cost almost nothing so there is no reason why you should share an instance
  • When would a developer need more than one data factory?
    • if a developer is doing active development on multiple branches of the same factory or is doing development on multiple factories.
  • Follow gitflow or some other process-based git development workflow. gitflow has advantages in that most developers understand it. Regardless, make sure your team understand the process and follows it for every change.
  • Train your developers on a "Day-in-the-Life of an ADF developer"
    • every bug or feature request gets a new branch.
    • development and testing is done on that branch.
    • when ready, issue a pull request to the team.
    • Even if you have one team member or don't care about code review, follow the PR process. This is always a best practice and only takes a few extra minutes of your time.

How to think about the collaboration and adf_publish branches

  • the collaboration branch is the SOURCE of all new branches when you branch from the ADF interface. It is therefore the branch that we will likely merge our PR into. This doesn't have to be the case, ADF can support very complex git workflows and branching strategies, but if you keep things simple this model has advantages. Regardless of the branching strategy for your other developers, it's best to keep ADF development simple.
    • the collaboration branch does not necessarily reflect what is ACTUALLY DEPLOYED. Always remember this. Instead, the collab branch is showing what the JSON in the repo contains
  • adf_publish: this contains the auto-generated ARM template code to deploy the data factory to Azure.
    • you rarely need to change this and when you do it's purposeful.
    • this contains the JSON for all of the linked services that you use, these MUST be published immediately so you can use them.
    • PartialArmTemplates folder will contain multiple json files for each linked service.
    • there is only ONE adf_publish branch. You can rename it but that will likely cause more confusion than it would be worth.

Development

  • keep all pipelines simple. Each pipeline should do only one "thing". Consider building small pipelines that can be strung together into larger pipelines. This is a simple OOP best practice.
    • I see too many people that are proud that they built these complex pipelines with literally a hundred activities. That's going to be a nightmare to maintain and test. Now, if you need to build a complex pipeline due to all of the joins you need to accomplish your task (as an example), that's fine, but think towards smaller pipelines.
  • each git commit and subsequent PR should be very atomic activities. If you need to build a pipeline with 5 new linked services, create those linked services (and the associated AKV stuff) before you code the pipeline. Linked service modifications will automatically change the adf_publish branch (see above for more). Checking these things in early and often will save you merge issues later. Remember, a pipeline is a single JSON file and anyone else working on that pipeline will modify the same json file. That means that file will be huge and difficult to maintain.
  • try to avoid having 2 people edit the same part of a data factory
    • save and publish often
    • merge often
    • make very granular changes. very granular!
    • after any merge, re-open browser