Use Amazon SageMaker Studio with a customized file system in Amazon EFS

Amazon SageMaker Studio is the newest web-based expertise for operating end-to-end machine studying (ML) workflows. SageMaker Studio gives a collection of built-in improvement environments (IDEs), which incorporates JupyterLab, Code Editor, in addition to RStudio. Information scientists and ML engineers can spin up SageMaker Studio personal and shared areas, that are used to handle the storage and useful resource wants of the JupyterLab and Code Editor functions, allow stopping the functions when not in use to avoid wasting on compute prices, and resume the work from the place they stopped.

The storage sources for SageMaker Studio areas are Amazon Elastic Block Retailer (Amazon EBS) volumes, which supply low-latency entry to person information like notebooks, pattern information, or Python/Conda digital environments. Nevertheless, there are a number of eventualities the place utilizing a distributed file system shared throughout personal JupyterLab and Code Editor areas is handy, which is enabled by configuring an Amazon Elastic File System (Amazon EFS) file system in SageMaker Studio. Amazon EFS supplies a scalable absolutely managed elastic NFS file system for AWS compute situations.

Amazon SageMaker helps routinely mounting a folder in an EFS quantity for every person in a website. Utilizing this folder, customers can share information between their very own personal areas. Nevertheless, customers can’t share information with different customers within the area; they solely have entry to their very own folder user-default-efs within the $HOME listing of the SageMaker Studio software.

On this submit, we discover three distinct eventualities that display the flexibility of integrating customized Amazon EFS with SageMaker Studio.

For additional data on configuring Amazon EFS in SageMaker Studio, discuss with Attaching a customized file system to a website or person profile.

Answer overview

Within the first situation, an AWS infrastructure admin needs to arrange an EFS file system that may be shared throughout the personal areas of a given person profile in SageMaker Studio. Which means that every person inside the area can have their very own personal house on the EFS file system, permitting them to retailer and entry their very own information and recordsdata. The automation described on this submit will allow new workforce members becoming a member of the information science workforce can rapidly arrange their personal house on the EFS file system and entry the mandatory sources to begin contributing to the continued challenge.

The next diagram illustrates this structure.

This situation gives the next advantages:

• Particular person information storage and evaluation – Customers can retailer their private datasets, fashions, and different recordsdata of their personal areas, permitting them to work on their very own tasks independently. Segregation is made by their person profile.
• Centralized information administration – The administrator can handle the EFS file system centrally, sustaining information safety, backup, and direct entry for all customers. By establishing an EFS file system with a personal house, customers can effortlessly monitor and preserve their work.
• Cross-instance file sharing – Customers can entry their recordsdata from a number of SageMaker Studio areas, as a result of the EFS file system supplies a persistent storage resolution.

The second situation is expounded to the creation of a single EFS listing that’s shared throughout all of the areas of a given SageMaker Studio area. Which means that all customers inside the area can entry and use the identical shared listing on the EFS file system, permitting for higher collaboration and centralized information administration (for instance, to share widespread artifacts). This can be a extra generic use case, as a result of there isn’t a particular segregated folder for every person profile.

The next diagram illustrates this structure.

This situation gives the next advantages:

• Shared challenge directories – Suppose the information science workforce is engaged on a large-scale challenge that requires collaboration amongst a number of workforce members. By establishing a shared EFS listing at challenge degree, the workforce can collaborate on the identical tasks by accessing and dealing on recordsdata within the shared listing. The information science workforce can, for instance, use the shared EFS listing to retailer their Jupyter notebooks, evaluation scripts, and different project-related recordsdata.
• Simplified file administration – Customers don’t must handle their very own personal file storage, as a result of they’ll depend on the shared listing for his or her file-related wants.
• Improved information governance and safety – The shared EFS listing, being centrally managed by the AWS infrastructure admin, can present improved information governance and safety. The admin can implement entry controls and different information administration insurance policies to take care of the integrity and safety of the shared sources.

The third situation explores the configuration of an EFS file system that may be shared throughout a number of SageMaker Studio domains inside the identical VPC. This permits customers from totally different domains to entry and work with the identical set of recordsdata and information, enabling cross-domain collaboration and centralized information administration.

The next diagram illustrates this structure.

This situation gives the next advantages:

• Enterprise-level information science collaboration – Think about a big group with a number of information science groups engaged on numerous tasks throughout totally different departments or enterprise models. By establishing a shared EFS file system accessible throughout the group’s SageMaker Studio domains, these groups can collaborate on cross-functional tasks, share artifacts, and use a centralized information repository for his or her work.
• Shared infrastructure and sources – The EFS file system can be utilized as a shared useful resource throughout a number of SageMaker Studio domains, selling effectivity and cost-effectiveness.
• Scalable information storage – Because the variety of customers or domains will increase, the EFS file system routinely scales to accommodate the rising storage and entry necessities.
• Information governance – The shared EFS file system, being managed centrally, will be topic to stricter information governance insurance policies, entry controls, and compliance necessities. This may also help the group meet regulatory and safety requirements whereas nonetheless enabling cross-domain collaboration and information sharing.

Conditions

This submit supplies an AWS CloudFormation template to deploy the primary sources for the answer. Along with this, the answer expects that the AWS account through which the template is deployed already has the next configuration and sources:

Discuss with Attaching a customized file system to a website or person profile for added conditions.

Configure an EFS listing shared throughout personal areas of a given person profile

On this situation, an administrator needs to provision an EFS file system for all customers of a SageMaker Studio area, creating a personal file system listing for every person. We will distinguish two use instances:

• Create new SageMaker Studio person profiles – A brand new workforce member joins a preexisting SageMaker Studio area and needs to connect a customized EFS file system to the JupyterLab or Code Editor areas
• Use preexisting SageMaker Studio person profiles – A workforce member is already engaged on a particular SageMaker Studio area and needs to connect a customized EFS file system to the JupyterLab or Code Editor areas

The answer offered on this submit focuses on the primary use case. We talk about how one can adapt the answer for preexisting SageMaker Studio area person profiles later on this submit.

The next diagram illustrates the high-level structure of the answer.

On this resolution, we use CloudTrail, Amazon EventBridge, and Lambda to routinely create a personal EFS listing when a brand new SageMaker Studio person profile is created. The high-level steps to arrange this structure are as follows:

1. Create an EventBridge rule that invokes the Lambda perform when a brand new SageMaker person profile is created and logged in CloudTrail.
2. Create an EFS file system with an entry level for the Lambda perform and with a mount goal in each Availability Zone that the SageMaker Studio area is situated.
3. Use a Lambda perform to create a personal EFS listing with the required POSIX permissions for the profile. The perform may also replace the profile with the brand new file system configuration.

Deploy the answer utilizing AWS CloudFormation

To make use of the answer, you’ll be able to deploy the infrastructure utilizing the next CloudFormation template. This template deploys three fundamental sources in your account: Amazon EFS sources (file system, entry factors, mount targets), an EventBridge rule, and a Lambda perform.

Discuss with Create a stack from the CloudFormation console for added data. The enter parameters for this template are:

• SageMakerDomainId – The SageMaker Studio area ID that will likely be related to the EFS file system.
• SageMakerStudioVpc – The VPC related to the SageMaker Studio area.
• SageMakerStudioSubnetId – One or a number of subnets related to the SageMaker Studio area. The template deploys its sources in these subnets.
• SageMakerStudioSecurityGroupId – The safety group related to the SageMaker Studio area. The template configures the Lambda perform with this safety group.

Amazon EFS sources

After you deploy the template, navigate to the Amazon EFS console and ensure that the EFS file system has been created. The file system has a mount goal in each Availability Zone that your SageMaker area connects to.

Notice that every mount goal makes use of the EC2 safety group that SageMaker created in your AWS account while you first created the area, which permits NFS site visitors at port 2049. The offered template routinely retrieves this safety group when it’s first deployed, utilizing a Lambda backed customized useful resource.

You can too observe that the file system has an EFS entry level. This entry level grants root entry on the file system for the Lambda perform that can create the directories for the SageMaker Studio person profiles.

EventBridge rule

The second fundamental useful resource is an EventBridge rule invoked when a brand new SageMaker Studio person profile is created. Its goal is the Lambda perform that creates the folder within the EFS file system and updates the profile that has been simply created. The enter of the Lambda perform is the occasion matched, the place you may get the SageMaker Studio area ID and the SageMaker person profile identify.

Lambda perform

Lastly, the template creates a Lambda perform that creates a listing within the EFS file system with the required POSIX permissions for the person profile and updates the person profile with the brand new file system configuration.

At a POSIX permissions degree, you’ll be able to management which customers can entry the file system and which recordsdata or information they’ll entry. The POSIX person and group ID for SageMaker apps are:

• UID – The POSIX person ID. The default is 200001. A legitimate vary is a minimal worth of 10000 and most worth of 4000000.
• GID – The POSIX group ID. The default is 1001. A legitimate vary is a minimal worth of 1001 and most worth of 4000000.

The Lambda perform is in the identical VPC because the EFS file system and it has hooked up the file system and entry level beforehand created.

Adapt the answer for preexisting SageMaker Studio area person profiles

We will reuse the earlier resolution for eventualities through which the area already has person profiles created. For that, you’ll be able to create an extra Lambda perform in Python that lists all of the person profiles for the given SageMaker Studio area and creates a devoted EFS listing for every person profile.

The Lambda perform needs to be in the identical VPC because the EFS file system and it has hooked up the file system and entry level beforehand created. It is advisable to add the efs_id and domain_id values as surroundings variables for the perform.

You’ll be able to embrace the next code as a part of this new Lambda perform and run it manually:

import json
import subprocess
import boto3
import os

sm_client = boto3.consumer('sagemaker')

def lambda_handler(occasion, context):
    
    # Get EFS and Area ID
    file_system=os.environ['efs_id']
    domain_id=os.environ['domain_id']    
    
    
    # Get Area person profiles
    list_user_profiles_response = sm_client.list_user_profiles(
        DomainIdEquals=domain_id
    )
    domain_users = list_user_profiles_response["UserProfiles"]
    
    # Create directories for every person
    for person in domain_users:

        user_profile_name = person["UserProfileName"]

        # Permissions
        repository=f'/mnt/efs/{user_profile_name}'
        subprocess.name(['mkdir', repository])
        subprocess.name(['chown', '200001:1001', repository])
        
        # Replace SageMaker person
        response = sm_client.update_user_profile(
            DomainId=domain_id,
            UserProfileName=user_profile_name,
            UserSettings={
                'CustomFileSystemConfigs': [
                    {
                        'EFSFileSystemConfig': {
                            'FileSystemId': file_system,
                            'FileSystemPath': f'/{user_profile_name}'
                        }
                    }
                ]
            }
        )

Configure an EFS listing shared throughout all areas of a given area

On this situation, an administrator needs to provision an EFS file system for all customers of a SageMaker Studio area, utilizing the identical file system listing for all of the customers.

To attain this, along with the conditions described earlier on this submit, you have to full the next steps.

Create the EFS file system

The file system must be in the identical VPC because the SageMaker Studio area. Discuss with Creating EFS file techniques for added data.

Add mount targets to the EFS file system

Earlier than SageMaker Studio can entry the brand new EFS file system, the file system will need to have a mount goal in every of the subnets related to the area. For extra details about assigning mount targets to subnets, see Managing mount targets. You may get the subnets related to the area on the SageMaker Studio console underneath Community. It is advisable to create a mount goal for every subnet.

Moreover, for every mount goal, it’s essential to add the safety group that SageMaker created in your AWS account while you created the SageMaker Studio area. The safety group identify has the format security-group-for-inbound-nfs-domain-id.

The next screenshot reveals an instance of an EFS file system with two mount targets for a SageMaker Studio area related to 2 subnets. Notice the safety group related to each mount targets.

Create an EFS entry level

The Lambda perform accesses the EFS file system as root utilizing this entry level. See Creating entry factors for added data.

Create a brand new Lambda perform

Outline a brand new Lambda perform with the identify LambdaManageEFSUsers. This perform updates the default house settings of the SageMaker Studio area, configuring the file system settings to make use of a particular EFS file system shared repository path. This configuration is routinely utilized to all areas inside the area.

The Lambda perform is in the identical VPC because the EFS file system and it has hooked up the file system and entry level beforehand created. Moreover, you have to add efs_id and domain_id as surroundings variables for the perform.

At a POSIX permissions degree, you’ll be able to management which customers can entry the file system and which recordsdata or information they’ll entry. The POSIX person and group ID for SageMaker apps are:

• UID – The POSIX person ID. The default is 200001.
• GID – The POSIX group ID. The default is 1001.

The perform updates the default house settings of the SageMaker Studio area, configuring the EFS file system for use by all customers. See the next code:

import json
import subprocess
import boto3
import os
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)
sm_client = boto3.consumer('sagemaker')

def lambda_handler(occasion, context):
    
    # Atmosphere variables
    file_system=os.environ['efs_id']
    domain_id=os.environ['domain_id']
    
    # EFS listing identify
    repository_name="shared_repository"
    repository=f'/mnt/efs/{repository_name}'
            
    # Add permissions to the brand new listing
    strive:
        subprocess.name(['mkdir -p', repository])
        subprocess.name(['chown', '200001:1001', repository])
    besides:
        print("Repository already created")
    
    # Replace Sagemaker area to allow entry to the brand new listing
    response = sm_client.update_domain(
        DomainId=domain_id,
        DefaultUserSettings={
            'CustomFileSystemConfigs': [
                {
                    'EFSFileSystemConfig': {
                        'FileSystemId': file_system,
                        'FileSystemPath': f'/{repository_name}'
                    }
                }
            ]
        }
    )
    logger.information(f"Up to date Studio Area {domain_id} and EFS {file_system}")
    return {
        'statusCode': 200,
        'physique': json.dumps(f"Created dir and modified permissions for Studio Area {domain_id}")
    }

The execution position of the Lambda perform must have permissions to replace the SageMaker Studio area:

{ 
"Model": "2012-10-17",
    "Assertion": [ 
        { 
        "Effect": "Allow", 
        "Action": [
            "sagemaker:UpdateDomain"
        ],
        "Useful resource": "*" 
        } 
    ]
}

Configure an EFS listing shared throughout a number of domains underneath the identical VPC

On this situation, an administrator needs to provision an EFS file system for all customers of a number of SageMaker Studio domains, utilizing the identical file system listing for all of the customers. The concept on this case is to assign the identical EFS file system to all customers of all domains which are inside the identical VPC. To check the answer, the account ought to ideally have two SageMaker Studio domains contained in the VPC and subnet.

Create the EFS file system, add mount targets, and create an entry level

Full the steps within the earlier part to arrange your file system, mount targets, and entry level.

Create a brand new Lambda perform

Outline a Lambda perform referred to as LambdaManageEFSUsers. This perform is answerable for automating the configuration of SageMaker Studio domains to make use of a shared EFS file system inside a particular VPC. This may be helpful for organizations that wish to present a centralized storage resolution for his or her ML tasks throughout a number of SageMaker Studio domains. See the next code:

import json
import subprocess
import boto3
import os
import sys

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

sm_client = boto3.consumer('sagemaker')

def lambda_handler(occasion, context):
    
    #Atmosphere variables
    event_domain_id =occasion["domain_id"]
    file_system=os.environ['efs_id']
    env_vpc_id =os.environ['vpc_id']
    
    #Occasion parameters 
    repository_name="shared_repository"
    repository=f'/mnt/efs/{repository_name}'
    domains =[]    

    # Checklist all SageMaker domains within the specified VPC
    response = sm_client.list_domains()
    all_domains = response['Domains']
    for area in all_domains:
        domain_id =area["DomainId"]
        information =sm_client.describe_domain(DomainId=domain_id)
        domain_vpc_id = information['VpcId']
        if domain_vpc_id ==env_vpc_id:
            domains.append(domain_id)
    
    # Create listing and add the permission
    strive:
        subprocess.name(['mkdir -p', repository])
        subprocess.name(['chown', '200001:1001', repository])
    besides:
        print("Repository already created")
    
    #Replace Sagemaker area
    if len(domains)>0:
        for domain_id in domains: 
            response = sm_client.update_domain(
                DomainId=event_domain_id,
                DefaultUserSettings={
                    'CustomFileSystemConfigs': [
                        {
                            'EFSFileSystemConfig': {
                                'FileSystemId': file_system,
                                'FileSystemPath': f'/{repository_name}'
                            }
                        }
                    ]
                }
            )
   
        logger.information(f"Up to date Studio for Domains {domains} and EFS {file_system}")
        return {
                'statusCode': 200,
                'physique': json.dumps(f"Created dir and modified permissions for Domains {domains}")
            }
    
    else:
        return {
            'statusCode': 400,
            'physique': json.dumps(f"VPC id of all of the domains {domain_vpc} is totally different than the vpc id configured {env_vpc_id}")
        }

The execution position of the Lambda perform must have permissions to explain and replace the SageMaker Studio area:

{ 
"Model": "2012-10-17",
    "Assertion": [ 
        { 
        "Effect": "Allow", 
        "Action": [
            "sagemaker:DescribeDomain",
            "sagemaker:UpdateDomain"
        ],
        "Useful resource": "*" 
        } 
    ]
}

Clear up

To scrub up the answer you carried out and keep away from additional prices, delete the CloudFormation template you deployed in your AWS account. Once you delete the template, you additionally delete the EFS file system and its storage. For extra data, discuss with Delete a stack from the CloudFormation console.

Conclusion

On this submit, we’ve got explored three eventualities demonstrating the flexibility of integrating Amazon EFS with SageMaker Studio. These eventualities spotlight how Amazon EFS can present a scalable, safe, and collaborative information storage resolution for information science groups.

The primary situation targeted on configuring an EFS listing with personal areas for particular person person profiles, permitting customers to retailer and entry their very own information whereas the administrator manages the EFS file system centrally.

The second situation showcased a shared EFS listing throughout all areas inside a SageMaker Studio area, enabling higher collaboration and centralized information administration.

The third situation explored an EFS file system shared throughout a number of SageMaker Studio domains, empowering enterprise-level information science collaboration and selling environment friendly use of shared sources.

By implementing these Amazon EFS integration eventualities, organizations can unlock the complete potential of their information science groups, enhance information governance, and improve the general effectivity of their data-driven initiatives. The mixing of Amazon EFS with SageMaker Studio supplies a flexible platform for information science groups to thrive within the evolving panorama of ML and AI.

Concerning the Authors

Irene Arroyo Delgado is an AI/ML and GenAI Specialist Options Architect at AWS. She focuses on bringing out the potential of generative AI for every use case and productionizing ML workloads, to realize prospects’ desired enterprise outcomes by automating end-to-end ML lifecycles. In her free time, Irene enjoys touring and mountain climbing.

Itziar Molina Fernandez is an AI/ML Advisor within the AWS Skilled Companies workforce. In her position, she works with prospects constructing large-scale machine studying platforms and generative AI use instances on AWS. In her free time, she enjoys exploring new locations.

Matteo Amadei is a Information Scientist Advisor within the AWS Skilled Companies workforce. He makes use of his experience in synthetic intelligence and superior analytics to extract helpful insights and drive significant enterprise outcomes for purchasers. He has labored on a variety of tasks spanning NLP, pc imaginative and prescient, and generative AI. He additionally has expertise with constructing end-to-end MLOps pipelines to productionize analytical fashions. In his free time, Matteo enjoys touring and studying.

Giuseppe Angelo Porcelli is a Principal Machine Studying Specialist Options Architect for Amazon Internet Companies. With a number of years of software program engineering and an ML background, he works with prospects of any measurement to grasp their enterprise and technical wants and design AI and ML options that make one of the best use of the AWS Cloud and the Amazon Machine Studying stack. He has labored on tasks in numerous domains, together with MLOps, pc imaginative and prescient, and NLP, involving a broad set of AWS providers. In his free time, Giuseppe enjoys enjoying soccer.