-
Notifications
You must be signed in to change notification settings - Fork 0
File System Configuration
This document contains specifications for the various file formats that Tank uses for its configuration and settings.
The folder configuration basically maps entities in Shotgun to locations on disk. Rather than using a single configuration file, the configuration is in the form of a "mini file system" which is expanded for each unit which is configured. Some folders in this mini file system represents a static folder on disk, that is essentially just copied across into the target area as folders are created. But more importantly, some folders represent a shotgun query, and therefore become multiple folders on disk.
The above image shows the configuration. It always starts with a folder named "project". This always represents the connected project in Shotgun. Below this level we have static folders which the folder creator will automatically create, for example the sequences folder.
However, inside the sequences folder, there is an sequence folder and a sequence.yml file. Whenever Tank detects a yml file with the same name as a folder, it will read the contents of the yml file and add the desired dynamic behaviour. In this case, the sequence.yml file contains the following
The structure underneath the project folder can consist of two types of items:
- Normal Folders and files - these are simply copied across to the target location.
- A folder with a yml file (having the same name as the folder) - This represents dynamic content - for example, it may be shot and shot.yml and when folders are created, this shot folder is expanded into a number of folders, one folder per shot.
The dynamic configuration setup expressed in the yml files currently supports the following modes:
- Dynamic fields based on a Shotgun Database Query. For example, this mode can be used to create a folder for every shot in a project.
- Dynamic fields based on a Shotgun List Field. For example, this mode can be used to create a folder for every value in the Shotgun List field Asset Type found on the Asset Entity.
- Deferred folders which are only executed during a second folder creation pass.
- A special folder, typically created in a second folder creation pass, which represents the current user.
When you want a dynamic folder which corresponds to a Shotgun query, use the following syntax in your yml file:
# the type of dynamic content
type: shotgun_entity
# the shotgun entity type to connect to
entity_type: Asset
# the shotgun field to use for the folder name
name: code
# shotgun filters to apply when getting the list of items
# this should be a list of dicts, each dict containing
# three fields: path, relation and values
# (this is std shotgun API syntax)
# any values starting with $ are resolved into path objects
filters: [ { "path": "project", "relation": "is", "values": [ "$project" ] } ]- Set type type field to be shotgun_entity
- The entity_type field should be set to the Shotgun entity from which we want to pull data.
- The name field is the name that should be given to each folder.
- You can use a single
field, like in the example above (e.g.
name: code). - You can use multiple fields in brackets (e.g.
name: "{asset_type}_{code}"). - If you want to include fields from other linked entities,
you can use the standard Shotgun dot syntax (e.g
name: "{sg_sequence.Sequence.code}_{code}").
- You can use a single
field, like in the example above (e.g.
- The filters field is a Shotgun Query. It follows the Shotgun API syntax relatively closely. It is a list of dictionaries, and each dictionary needs to have the keys path, relation and values. For shotgun entity links, you can use the $syntax to refer to a parent folder in the configuration - this is explained more in depth in the examples below.
A shotgun_entity folder also supports an optional flag to control whether the folder creation process should try recurse down into it:
# recurse down from parent folder
create_with_parent: trueThis setting is optional and set to false by default. If you set it to true, Tank will try to
create folders for any child entities it finds. This setting is useful whenever there is nesting
inside shotgun data structures. For example, Sequences are typically parents to Shots in the file
system. If you want Shots to be created whenever their parent sequence is created,
set create_with_parent to true for the shot. The default setting in Tank is false,
meaning that if you create folders for a sequence, it wont create shot folders automatically.
Normally when you define the folder name (e.g. {code}_{sg_extra_field}), Tank requires all fields
to have values in Shotgun. For example, if the sg_extra_field is blank, an error message will be
generated. If you have a field that is sometimes populated and sometimes not, you can mark it as
optional. This means that tank will include it if it has a value, and exclude it if the value
is blank.
You define optional fields using square brackets, like this: {code}[_{sg_extra_field}].
This would generate the following folder names:
- If
codeis AAA andsg_extra_fieldis extra, the folder name will beAAA_extra - If
codeis AAA andsg_extra_fieldis blank, the folder name will beAAA
Below are a collection of examples showing how to use the filters syntax:
To find all shots which belong to the current project and are in progress, use the syntax below. Note that the Shotgun Shot Entity has a link field named project which connects a shot to a project. We want to make sure that we only create folders for the shots that are associated with the current project. Because there is a project level higher up in the configuration file system, we can refer to this via the $syntax - tank will automatically create this to a shotgun entity link reference.
entity_type: Shot
filters:
- { "path": "project", "relation": "is", "values": [ "$project" ] }
- { "path": "status", "relation": "is", "values": [ "ip" ] } If you have a Sequence folder higher up the tree and want to create folders for all shots which belong to that sequence, you could create the following filters:
entity_type: Shot
filters:
- { "path": "project", "relation": "is", "values": [ "$project" ] }
- { "path": "sg_sequence", "relation": "is", "values": [ "$sequence" ] }To find all assets use this syntax:
entity_type: Asset
filters: [ { "path": "project", "relation": "is", "values": [ "$project" ] } ]When you want a dynamic folder which corresponds to all the items in a Shotgun list field, use the following syntax in your yml file:
# the type of dynamic content
type: "shotgun_list_field"
# the shotgun entity type to connect to
entity_type: "Asset"
# only create for values which are used in this project.
# this is optional and will be set to false if not specified.
skip_unused: false
# the shotgun field to use for the folder name
field_name: "{sg_asset_type}_type"- Set type type field to be shotgun_list_field
- The entity_type field should be set to the Shotgun entity from which we want to pull data.
- The field_name field should be set to the shotgun field from which we should pull the data. This shotgun field must be of type list. You can use expressions if you want to add static text alongside the dynamic content.
- The optional skip_unused will prevent the creation of directories for list field values which are not used.
This folder type can be useful if you for example want to create one folder for every asset type in Shotgun. Asset types are just a list field in Shotgun, and this folder config type makes it possible to define a layer in the file system that reflects those asset type listings.
Please note that once folders have been created on disk, we strongly advise not to change the value (e.g. the asset type) on the associated data.
The pipeline step folder represents a pipeline step in Shotgun.
# the type of dynamic content
type: "shotgun_step"
# the shotgun field to use for the folder name. This field needs to come from a step entity.
name: "short_name" Note that you can use name expressions, just like you can with the Shotgun entity described above. The node will look at its parent, grand parent etc. until a Shotgun Entity Folder configuration is found. This entity folder will be associated with the step and will be used to determine which steps to create. (If you want to create a top level folder with pipeline steps, just use the shotgun entity node and set the associated type to be step!).
By default, the step folder will try to create all the relevant steps for a particular entity automatically. For example, if the folder creation is triggered for a shot which has a step node associated, steps folders will automatically be created.
You can, however, turn this off by using the following syntax:
# recurse down from parent folder
create_with_parent: falseAdding this setting to the configuration means that no step folders will be created when a shot folder is created. Instead, only when you run the folder creation for a task, folders will be created. This can be useful if you want to configure user sandboxes and other structure which is created just before work starts.
The task folder represents a task in Shotgun.
# the type of dynamic content
type: "shotgun_task"
# the shotgun field to use for the folder name. This field needs to come from a task entity.
name: "content" Note that you can use name expressions, just like you can with the Shotgun entity described above. The node will look at its parent, grand parent etc. until a Shotgun Entity Folder configuration is found. This entity folder will be associated with the task and will be used to determine which task folders to create.
By default, the task folder will not recurse down - for example, if the folder creation is triggered for a shot which has a task node associated, task folders will not automatically be created. Instead, these folders would only be created when the folder creation is executed for that task.
You can, however, turn on creation so that tasks are created with their parent entity by using the following syntax:
# recurse down from parent folder
create_with_parent: trueThe current user folder is a special construct that let's you set up user work areas for different users. The configuration file needs to include the following options:
# the type of dynamic content
type: "user_workspace"
name: "login"- Set type type field to be user_workspace
- The name field is the name that should be given to a user folder. It must consist of a combination
of fields fetched from the Shotgun
HumanUserentity.- You can use a single field, like in the example above (e.g.
name: login). - You can use multiple fields in brackets (e.g.
name: "{firstname}_{surname}"). - If you want to include fields from other linked entities,
you can use the standard Shotgun dot syntax (e.g
name: "{sg_group.Group.code}_{login}").
- You can use a single field, like in the example above (e.g.
The current user folder is created as a deferred folder by default, meaning that it will only be executed
when a second folder creation pass is requested via the optional engine parameter in the create folders
method in the Tank API. Typically, this method is executed by Tank's various application launchers, just
prior to starting up an application, meaning that a user folder is created on disk just before the
application starts up.
Most folder types support a deferred flag, which is false by default.
# only create this folder when maya or nuke launches
defer_creation: ["tk-maya", "tk-nuke"]
# create this folder when any application launches, but not when normal folder
# creation runs
defer_creation: trueThis flag makes it possible to split the folder creation in half - one part that runs in a first "global" pass and a second pass that runs at a later point. Typically, this pass is associated with the engine launching (although it does not happen automatically) and allows for a user to create folders just before engine startup. This allows for two primary workflows:
- Workspaces - application specific folder setups - can be created just before an application launches.
- User folders - a user folder is created just before application launch. The user folder config construct (described above) is deferred by default.
If you want a normal, static folder to be created just in time when an application (like maya) launches, just create a config yml file named the same as the folder and add the following:
# type of content
type: "static"
# only create this folder for maya
defer_creation: "tk-maya"
Files that are placed in the schema scaffold will be copied across into the target area as part of the folder creation. This copy process is handled by a core hook so for example permissions handling can be customized for a project or studio.
Sometimes it can be useful to exclude certain files and folders from being copied across as
part of the folder creation. For example, if you store your folder creation configs
in git or svn, you will have .git and .svn folders that you don't want to copy
to each shot or asset folder. If there are files which you do not want to have copied, a file named
ignore_files can be placed in config/core/schema folder inside the project configuration.
This file should contain glob style
patterns to define files not to copy. Each pattern should be on a separate line.
# This is a good example of a standard ignore_files file
.svn # no svn temp files to be copied across at folder creation time
.git # no git temp files to be copied across at folder creation time
.DS_Store # no mac temp files to be copied across at folder creation time
You can also use wild cards, for example if you need to exclude all files with the
tmp extension, just add a *.tmp line to the file.
Shot and Asset folders often need to be created with special permissions and parameters. Sometimes this is a simple as setting permission bits during the folder creation, and sometimes it may be as complex as sending a remote request to a special folder creation server that will create the folders with the appropriate credentials, groups and permissions.
It is also common that folders on different levels in the file system tree need to have different permissions; a work area folder is typically writeable for everybody, whereas a shot folder may have much stricter permissions.
Tank allows for customization of the folder creation via a single hook. As the folder creation API call is traversing the folder configuration and deciding which folders should be created, it builds up a list of items that need to be created. These items can be both files and folders. As the final step of the folder creation, this list is past into a hook to handle the actual folder processing. Alongside, the actual file names of the items to be created, the hook also passes the configuration yml metadata files from the folder creation config. This is handy if you for example want to control permissions of the created files - you can then add your own custom permissions settings in the yml files in the folder creation config, and this configuration data will be passed into the hook at folder creation time.
A simple folder creation hook could look something like this:
class ProcessFolderCreation(Hook):
def execute(self, items, preview_mode, **kwargs):
"""
The default implementation creates folders recursively using open permissions.
This hook should return a list of created items.
Items is a list of dictionaries. Each dictionary can be of the following type:
Standard Folder
---------------
This represents a standard folder in the file system which is not associated
with anything in Shotgun. It contains the following keys:
* "action": "folder"
* "metadata": The configuration yaml file for this item
* "path": path on disk to the item
Entity Folder
-------------
This represents a folder in the file system which is associated with a
shotgun entity. It contains the following keys:
* "action": "entity_folder"
* "metadata": The configuration yaml file for this item
* "path": path on disk to the item
* "entity": Shotgun entity link dict with keys type, id and name.
File Copy
---------
This represents a file copy operation which should be carried out.
It contains the following keys:
* "action": "copy"
* "metadata": The configuration yaml file associated with the directory level
on which this object exists.
* "source_path": location of the file that should be copied
* "target_path": target location to where the file should be copied.
File Creation
-------------
This is similar to the file copy, but instead of a source path, a chunk
of data is specified. It contains the following keys:
* "action": "create_file"
* "metadata": The configuration yaml file associated with the directory level
on which this object exists.
* "content": file content
* "target_path": target location to where the file should be copied.
"""
# set the umask so that we get true permissions
old_umask = os.umask(0)
folders = []
try:
# loop through our list of items
for i in items:
action = i.get("action")
if action == "entity_folder" or action == "folder":
# folder creation
path = i.get("path")
if not os.path.exists(path):
if not preview_mode:
# create the folder using open permissions
os.makedirs(path, 0777)
folders.append(path)
elif action == "copy":
# a file copy
source_path = i.get("source_path")
target_path = i.get("target_path")
if not os.path.exists(target_path):
if not preview_mode:
# do a standard file copy
shutil.copy(source_path, target_path)
# set permissions to open
os.chmod(target_path, 0666)
folders.append(target_path)
elif action == "create_file":
# create a new file based on content
path = i.get("path")
parent_folder = os.path.dirname(path)
content = i.get("content")
if not os.path.exists(parent_folder) and not preview_mode:
os.makedirs(parent_folder, 0777)
if not os.path.exists(path):
if not preview_mode:
# create the file
fp = open(path, "wb")
fp.write(content)
fp.close()
# and set permissions to open
os.chmod(path, 0666)
folders.append(path)
else:
raise Exception("Unknown folder hook action '%s'" % action)
finally:
# reset umask
os.umask(old_umask)
return folders
The Tank templates file is one of the hubs of the Tank configuration. There is always one of these files per project and it resides inside the tank/config/core folder.
This file contains definitions for templates and their keys.
A key is a dynamic field we defined. It can be a name, a version number, a screen resolution, a shot name etc. Keys are configured with types, so we can define that a key should be a string or an int for example. They are also formatted, so we can define that a string should only contain alpha numeric characters, or that all integers should be padded with eight zeroes.
A template is a dynamic path. An example of a template is
shots/{shot}/publish/{name}.{version}.ma. This template could for represent maya publishes for a
shot - the bracketed fields are keys.
The templates file is divided into three sections: keys, paths and strings.
Keys define what values are acceptable for fields. In the template config file keys are defined in the form:
key_name:
type: key_type
option: option_value
option: option_valueKey type is either str, int, or sequence. Str keys are keys whose values are strings,
int keys are keys whose values are integers, and sequence keys are keys whose values are
sequences of integers.
In addition to specifying the type, you can also specify additional options. The following options exist:
-
default: default_value- Value used if no value was supplied. This can happen if you are using the Tank API and trying to resolve a set of field values into a path for example. -
choices: [choice1, choice2, etc]- An enumeration of possible values for this key. -
exclusions: [bad1, bad2, etc]- An enumeration of forbidden values for this key. If key is of type sequence, frame spec values cannot be invalidated with this setting. -
length: 12- This key needs to be of an exact length. -
alias: new_name- Provides a name which will be used by templates using this key rather than the key_name. For example if you have two concepts of a version number, one is four zero padded because that is how the client wants it, and one is three zero padded because that how it is handled internally - in this case you really want both keys named "version" but this is not really possible since key names need to be unique. In this case you can create an alias. See one of the examples below for more information. -
filter_by: alphanumeric- Only works for keys of type string. If this option is specified, only strings containing alphanumeric values (a-z, A-Z and 0-9) will be considered valid values. -
format_spec: "04"- Only works for keys of type int or sequence. This setting means that the int or sequence number will be zero padded. Specifying "04" like in the example will result in four zeroes padding. Specifying "03" would result in three zeroes padding, etc. -
shotgun_entity_type- When used in conjunction with theshotgun_field_nameoption, will cause contexts to query shotgun directly for values. This allows using values from fields not seen in the folder structure to be used in file names. -
shotgun_field_name- Only used in conjunction withshotgun_entity_type. -
abstract- Denotes that the field is abstract. Abstract fields are used when a pattern is needed to describe a path - for example image sequences (%04d) or stereo (%V). Abstract fields require a default value.
A name that defaults to "comp" and that is alphanumeric
name:
type: str
default: "comp"
filter_by: alphanumeric
nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{name}.v{version}.nkA version number that would match numbers such as 002, 102, 034, 12341
version:
type: int
format_spec: "03"A typical stereo eye setup. The eye field is either L or R, but when used in software, it is often referred to in a generic, abstract fashion as %V. Since %V does not really refer to a file name but rather a collection of files, we set the abstract flag. Abstract fields need to have a default value that is pulled in whenever the abstract representation is being requested.
eye:
type: str
choices: ["L", "R", "%V"]
default: "%V"
abstract: true
nuke_shot_render_stereo: sequences/{Sequence}/{Shot}/{Step}/work/images/{Shot}_{name}_{eye}_v{version}.{SEQ}.exrImage sequences are abstract by definition and they have a default value set to %0Xd unless otherwise specified. The below sequence spec would identify frame numbers such as 0001, 1234 and 12345.
SEQ:
type: sequence
format_spec: "04"
nuke_shot_render_stereo: sequences/{Sequence}/{Shot}/{Step}/work/images/{Shot}_{name}_{channel}_{eye}_v{version}.{SEQ}.exrTwo definitions of version number that can both be used by code that expects a key which is named "version". This is useful if you have two Tank apps that both need a version field but you want these version field to be formatted differently.
nuke_version:
type: int
format_spec: "03"
alias: version
maya_version:
type: int
format_spec: "04"
alias: version
# nuke versions are using numbers on the form 003, 004, 005
# the nuke publish app requires a field called {version}
# however {nuke_version} is a valid replacement for {version}
# because it has an alias defined
nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{name}.v{nuke_version}.nk
# maya versions are using numbers on the form 0004, 0005, 0006
maya_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/maya/{name}.v{maya_version}.ma
This is useful when you want to add shotgun fields to for example a file name. For example, if you wanted to include the user name in a file name, you could use the following definition:
current_user_name:
type: str
shotgun_entity_type: HumanUser
shotgun_field_name: login
nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{current_user_name}_{name}.v{version}.nkWhen a Tank app populates all the context fields (via the context.as_template_fields() method, it
will populate the higher level fields Shot, Sequence and Step automatically. It will also
scan through all fields which have shotgun_entity_type defined (like our current_user_name field
above). If the Shotgun Entity is defined in the context, it will be able to automatically resolve the
value. The current user is always tracked in the context, and in the above example, it would also be
possible to pull data from fields on Shot, Sequence and Step since these are defined as part of the
higher level path and therefore part of the context. However, trying to refer to an Asset entity in
a field wouldn't work in the above example since Tank would have no way of knowing which asset in Shotgun to pull the data from.
A string value with two valid values:
maya_file_extension:
type: str
choices: ["ma", "mb"]A string field for which the value "assets" is not allowed. This is useful if you for example have two a folder which contains folders for all the sequences for a project alongside with a single "assets" folder where all the assets are kept:
project
|--- sequence1
|--- sequence2
|--- sequence3
\--- assets
In order for tank to correctly understand that the assets folder is not just another sequence, we can define that "assets" is not a valid value for the sequence template.
sequence:
type: str
exclusions: ["assets"]The exclusions field above allows us to define two templates that both correctly resolves:
sequence_work_area: {sequence}/{shot}/work
asset_work_area: assets/{asset}/work All paths consist of at least a name and a definition, where the definition is a combination of key names in brackets intersperced with non-key values representing a path. For example, a definition for a shot work file might look like:
shot_work: sequences/{Sequence}/{Shot}/{Step}/work/{Shot}.v{version}.maWith Sequence, Shot, Step and version being keys defined in the same template file.
An alternate form for paths is:
path_name:
definition: path_definition
option: option_valueThis form is required if any optional attributes are to be defined. Currently the only optional attribute root_name, which can be used to specify a project root for a path in a project that has multiple roots.
root_name: name_of_project_rootStrings are similar to paths in that they must include a name and definition, which can be supplied in the simple form:
string_name: string_definitionString definitions are templates consisting of key names and other values which together resolve to a string rather than a file system path. An example might the name used in Shotgun for a publish:
maya_publish_sg_name: "Maya publish, {name}, v{version}"With name and version as key names defined in the same file.
Optional sections can be defined using square brackets:
shot_work: sequences/{Shot}/work/{Shot}.[v{version}.]maThe optional section must contain at least one key. If the path is resolved with no value for the key(s) in an optional section, the path will resolve as if that section did not exist in the definition. The example above can be thought of as two templates baked into a single definition;
shot_work: sequences/{Shot}/work/{Shot}.v{version}.ma
shot_work: sequences/{Shot}/work/{Shot}.maAs you pass in a dictionary of fields, Tank will choose the right version of the template depending on the values:
>>> template = tk.templates["shot_work"]
>>> template.apply_fields({"Shot":"ABC_123", "version": 12}
/project/sequences/ABC_123/work/ABC_123.v12.ma
>>> template.apply_fields({"Shot":"ABC_123"}
/project/sequences/ABC_123/work/ABC_123.ma