Skip to content

Lab021.md

Latest commit

 

History

History
56 lines (43 loc) · 5.46 KB

Lab021.md

File metadata and controls

56 lines (43 loc) · 5.46 KB
Raw

Lab 021: Version Control Best Practices

This lab is conversational.

Synapse is browser-based development

  • Restart your browser often
    • Occasionally a complex pipeline won't work correctly, even after you know you've fixed it. Simply save and commit your work, close your browser and start over. The DOM seems to get corrupted. This is less of an issue in recent years.
  • Use a separate browser for Synapse development
    • If you use chrome as your daily-driver, consider using chromium, google chrome canary, firefox, edge, etc, for Synapse work. You don't want to have to restart your browser due to Synapse issues and lose your email windows, etc. Likewise, I know it shouldn't happen, but it seems like Synapse sees more issues when I share the browser with other non-Synapse activities.
  • Whenever something doesn't appear to be working as expected, restart the browser. This includes cases where Serverless queries seem to be taking MUCH LONGER than they should be. They are likely running ok but the browser webhook is having issues.

Version Control

  • Never work without git. Never.
  • Consider using gitflow.

Should you have one one repo for:

  • each factory
  • all ADF factories
  • shared repo with other data projects ???

It really doesn't matter and is more a function of your development culture and release lifecycle. Some teams have monolith repos, some have individual items in their own repos. Since you generally have one DevOps pipeline per repo, you will have to deploy all items within that repo unless you segregate by folder structure. This would seem to indicate that the best practice is one repo/factory.

Team-based Synapse development

  • by default Synapse authors directly against the Synapse services. Don't do this due to the possibility of DOM corruption mentioned above. Use git.
  • Follow gitflow or some other process-based git development workflow. gitflow has advantages in that most developers understand it. Regardless, make sure your team understands the process and follows it for every change.
  • Train your developers on a "Day-in-the-Life of an Synapse developer"
    • every bug or feature request gets a new branch.
    • development and testing is done on that branch.
    • when ready, issue a pull request to the team.
    • Even if you have one team member or don't care about code review, follow the PR process. This is always a best practice and only takes a few extra minutes of your time.
  • CI/CD guidance

How to think about the collaboration and adf_publish branches

  • the collaboration branch is the SOURCE of all new branches when you branch from the ADF interface. The default is master but I advise always changing it to collaboration. This results in a little mental nudge to make sure developers aren't doing development directly in master. collaboration is therefore the branch that we will likely merge our PR into. This doesn't have to be the case, Synapse can support very complex git workflows and branching strategies, but if you keep things simple this model has advantages. Regardless of the branching strategy for your other developers, it's best to keep Synapse development simple.
    • the collaboration branch does not necessarily reflect what is ACTUALLY DEPLOYED. Always remember this. Instead, the collab branch is showing what the JSON in the repo contains. You still have to publish
  • workspace_publish: this contains the auto-generated ARM template code to deploy the data factory to Azure.
    • you rarely need to change this and when you do it's purposeful.
    • this contains the JSON for all of the linked services that you use, these MUST be published immediately so you can use them. Unless you use key vault, then you do NOT need to immediately publish.
    • PartialArmTemplates folder will contain multiple json files for each linked service.
    • there is only ONE workspace_publish branch. You can rename it but that will likely cause more confusion than it would be worth.

Development

  • keep all pipelines simple. Each pipeline should do only one "thing". Consider building small pipelines that can be strung together into larger pipelines. This is a simple OOP best practice.
    • I see too many people that are proud that they built these complex pipelines with literally a hundred activities. That's going to be a nightmare to maintain and test. Now, if you need to build a complex pipeline due to all of the joins you need to accomplish your task (as an example), that's fine, but think towards smaller pipelines.
  • each git commit and subsequent PR should be very atomic activities. If you need to build a pipeline with 5 new linked services, create those linked services (and the associated AKV stuff) before you code the pipeline. Linked service modifications will automatically change the workspace_publish branch (see above for more). Checking these things in early and often will save you merge issues later. Remember, a pipeline is a single JSON file and anyone else working on that pipeline will modify the same json file. That means that file will be huge and difficult to maintain.
  • try to avoid having 2 people edit the same part of a data factory
    • save and publish often
    • merge often
    • make very granular changes. very granular!
    • after any merge, re-open browser