Forge

EC2 fleets made easy

---

Project maintained by carsdotcom Hosted on GitHub Pages — Theme by mattgraham

Home

---

Environmental Yaml (setup by admins)

Each forge job requires an environment created. This includes an environment yaml and the default user data scripts (optional).

Setup

https://github.com/carsdotcom/cars-forge/blob/main/examples/env_yaml_example/example.yaml

Option 1: Copy files to the config folder

1. Referring to the example yaml, create your environment yaml file.
2. Create your default user data script (optional). An example can be found here.
   • In the user_data parameter in the environment yaml, you can specify what the default user data script should be if the user does not provide one.
   • echo "$(cat /root/.ssh/authorized_keys | sed 's/^.*ssh-rsa/ssh-rsa/')" > /root/.ssh/authorized_keys is needed in all user data scripts because Forge runs all commands as root.
3. Run forge configure -h
   • This will output the location of the configure folder.
   • Example: Configure env yaml and place it in /home/ec2-user/.local/lib/python3.7/site-packages/forge/config/
4. Create a new folder under the config folder.
   • E.g.

      mkdir -p /home/ec2-user/.local/lib/python3.7/site-packages/forge/config/example`
     

5. Copy your environment yaml and user data script to the new folder.

Option 2: Use forge configure

1. Run forge configure and fill in all the parameters.
2. Create your default user data script (optional). An example can be found here.
   • In the user_data parameter in the environment yaml, you can specify what the default user data script should be if one is not provided.
   • echo "$(cat /root/.ssh/authorized_keys | sed 's/^.*ssh-rsa/ssh-rsa/')" > /root/.ssh/authorized_keys is needed in all user data scripts because Forge runs all commands as root.
3. Run forge configure -h
   • This will output the location of the configure folder.
   • Example: Configure env yaml and place it in /home/ec2-user/.local/lib/python3.7/site-packages/forge/config/
4. Locate the environment under the config folder. The folder name will be the same as the name you used when you ran forge configure.
5. Copy your user data script to the new folder.

Environmental parameters (setup by admins)

• additional_config - Additional configuration options allowed in user config files
  • Needs four parameters:
    • name: name of the parameter
    • type: data type
    • default: default value if one isn’t given.
    • constraints: options they must choose
    • error: error message if a given value is the wrong type or not within constraints
  • E.g. ```yaml additional_config:
    • name: pip type: list default: [] constraints: [] error: “”
    • name: version type: float default: 2.3 constraints: [2.3, 3.0, 3.1] error: “Invalid Spark version. Only 2.3, 3.0, and 3.1 are supported.” ```
• aws_az - The AWS availability zone where Forge will create the EC2 instance. Currently, Forge can run only in one AZ
• aws_profile - AWS CLI profile to use
• aws_security_group - AWS Security Group for the instance
• aws_subnet - AWS subnet where the EC2s will run
• default_ratio - Override the default ratio of RAM to CPU if the user does not provide one. Must be a list of the minimum and maximum.
  • default is [8, 8]
• ec2_amis - A dictionary of dictionaries to store AMI info.
  • Needs three parameters: - ami id - minimum disk size - device name
  • E.g.

      ec2_amis:
    single:
      ami: ami-abcdefghi12345678
      disk: 30
      disk_device_name: /dev/sda1
    cluster:
      ami: ami-12345678abcdefghi
      disk: 30
      disk_device_name: /dev/xvda
    single_gpu:
      ami: ami-abcdefghi00000000
      disk: 90
      disk_device_name: /dev/sda1
    

• ec2_key - The key pair name used to ssh into the EC2. This will be stored in forge_pem_secret
• ec2_max - Override the default maximum amount of RAM a single instance can have. The default is 768.
• excluded_ec2s - List of EC2 types to not consider when creating instances. You can use strings with one or more wild cards, represented by an asterisk (*)
  • E.g.

      excluded_ec2s: ["*g.*", "gd.*", "*metal*", "g4ad*"]
    

• forge_env - Name of the Forge environment. The user will refer to this in their yaml.
• forge_pem_secret - The secret name where the ec2_key is stored
• tags - Tags to apply to instances created by Forge. Follows the AWS tag format.
  • Forge also exposes all string, numeric, and some extra variables from the combined user and environmental configs that will be replaced at runtime by the matching values (e.g. {name} for job name, {date} for job date, etc.) See the variables page for more details.
  • E.g. ```yaml tags:
    • Key: “Name” Value: “{name}-{market}-{task}-{date}”
    • Key: “Application” Value: “{name}”
    • Key: “User” Value: “{user}” ```
• user_data - Paths to the default user_data scripts. Can be overwritten by the user.
  • Forge also exposes all string, numeric, and some extra variables from the combined user and environmental configs that will be replaced at runtime by the matching values (e.g. {name} for job name, {date} for job date, etc.) See the variables page for more details.
  • Note Forge uses the root user to rsync and run. One way to allow this is to add the below to the user data.

      #!/bin/bash
      set -x
      echo "$(cat /root/.ssh/authorized_keys | sed 's/^.*ssh-rsa/ssh-rsa/')" > /root/.ssh/authorized_keys