CTF Writeup: CICD GOAT | Part 1

, , ,

banner1.png

Hello y’all, i was recently learning about DevSecOps and came across this CICD GOAT that is a CTF challenge platform and as its essential part of DevSecops i decided to give a try where i tried to solve some challenges by myself and also took help from other resources as purpose is to learn CICD security therefore i’ll try to write this blog as comprehensive as possilbe.

CICD GOAT?

its a CTF platform developed by cider-secuirty-research that cover OWASP Top 10 CICD(Continuous Integration and Continuous Deployment (or Continuous Delivery)) vulnerabilities/misconfigurations that being made deribately vulnerable, where they covered compromising the security of Build Server, Pipeline, Environment and Secrets.

Installation

Follow the steps from here, or

Make sure to update docker and compose.

Linux & Mac

1
2
curl -o cicd-goat/docker-compose.yaml --create-dirs https://raw.githubusercontent.com/cider-security-research/cicd-goat/main/docker-compose.yaml
cd cicd-goat && docker compose up -d

Windows (Powershell)

1
2
3
4
mkdir cicd-goat; cd cicd-goat
curl -o docker-compose.yaml https://raw.githubusercontent.com/cider-security-research/cicd-goat/main/docker-compose.yaml
get-content docker-compose.yaml | %{$_ -replace "bridge","nat"}
docker compose up -d

Prerequisite

It must to have knowledge about DevSecOps and CICD Pipeline i’ll recommend you to go through this path/learning materials:

DevSecOps and Secure CICD

https://tryhackme.com/path/outline/devsecops
https://www.youtube.com/playlist?list=PLjNII-Jkdjfz5EXWlGMBRk63PC8uJsHMo

OWASP Top 10 CI/CD Security Risks

https://owasp.org/www-project-top-10-ci-cd-security-risks/

Access Credentials

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CTFd at http://localhost:8000
	- Username: alice
    - Password: alice

Jenkins at http://localhost:8080
	- Username: alice
	- Password: alice

Gitea at http://localhost:3000
	- Username: thealice
	- Password: thealice

GitLab at http://localhost:4000
	- Username: alice
	- Password: alice1234

Note

This CTF is designed considering we have already gained our foothold as or compromised developer position or we are the developer with evil mind set 💀.
as we will be having read/write permission in VCS(Version Control System) i.e. Gitea here and read permission in Jenkin to perfrom the attacks.

so initial diagram of entire platform is like

diagram1

compromised platform diagram:

diagram2

Challenges - Easy

White Rabbit

Description

white-rabbit.png

Hint 1 : Try to trigger a pipeline through the repository.
Hint 2 : How can you access credentials using the Jenkinsfile?

From description its clear that we have to get the secret i.e. flag1 by triggering a pipeline through the repository Wonderland/white-rabbit using our access as alice stored in the Jenkins credential store.

after logging in Gitea as alice we can see there some repositories where our use one is white-rabbit

gitea

and then jenkinfile that contain the entire script for automate the pipeline.

Jenkinsfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
pipeline {
    agent any
    environment {
        PROJECT = "src/urllib3"
    }

    stages {
        stage ('Install_Requirements') {
            steps {
                sh """
                    virtualenv venv
                    pip3 install -r requirements.txt || true
                """
            }
        }

        stage ('Lint') {
            steps {
                sh "pylint ${PROJECT} || true"
            }
        }

        stage ('Unit Tests') {
            steps {
                sh "pytest"
            }
        }

    }
    post { 
        always { 
            cleanWs()
        }
    }
}

If you are not familiar with syntax then you can refer this link

there are several stages in the script that seems to no use for us, as we have are tasked to get the flag1 from Jenkin credential storage, i came across this doc that contain this section that explain the usage of credentials i.e. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline {
    agent {
        // Define agent details here
    }
    environment {
        AWS_ACCESS_KEY_ID     = credentials('jenkins-aws-secret-key-id')
        AWS_SECRET_ACCESS_KEY = credentials('jenkins-aws-secret-access-key')
    }
    stages {
        stage('Example stage 1') {
            steps {
                //
            }
        }
        stage('Example stage 2') {
            steps {
                //
            }
        }
    }
}

we can change our Jenkingfile to get the flag1 secret that may misconfigured and visible to other job or user as well.

We can do it manually or using git utility, so i created a new branch and after making changes to script pushed the file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
pipeline {
    agent any
    environment {
        PROJECT = "src/urllib3"
        SECRET = credentials("flag1")
    }

    stages {
        stage ('Printing the secret') {
            steps {
                sh """
						echo $SECRET
                """
            }
        }
    }
    post { 
        always { 
            cleanWs()
        }
    }
}

jenkin_edit0

and as we can see it Gitea repo there is another branch created with name zr0x with our committed changes

jenkin_edit_1
jenkin_edit_2

now we have to trigger the pipeline by creating the PR(Pull Request) to the main branch so that we can get the secret on jenkin console as we have read access to it

jenkin_edit_3

now we’ll gonna see pull request in Jenkin that automaticlly have gone through pipeline stages where we can look at console for secret

jenkin_edit_5

Unfortunately, its not visible or being masked by jenkin because of this Masks Passwords plugin features

credential_masking.png

We circumvent printing it in encoded format i.e. echo $SECRET | base64

jenkin_edit_6

and we got the flag.

Learning Takeaway

So what was vunlerability/misconfiguration and why it was possible to perform the attack.

1. CICD-SEC-4: Poisoned Pipeline Execution (PPE):

Poisoned Pipeline Execution (PPE) risks refer to the ability of an attacker with access to source control systems - and without access to the build environment, to manipulate the build process by injecting malicious code/commands into the build pipeline configuration, essentially ‘poisoning’ the pipeline and running malicious code as part of the build process.

we did able to make changes to the pipeline i.e. Jenkinfile by forking/cloning with unkown branch and successfully created a PR that triggered the pipeline end up getting secret from the jenkin credential store and as we did it directly to the repository file its called D-PPE

D-PPE

Instead, the preferred configuration or security practice would be to prevent the pipeline from being modified and triggered by pull requests originating from unknown branches. Only trusted or protected branches should be allowed to trigger the pipeline.

Mitigation

PPI-Mitigations.png

2. CICD-SEC-6: Insufficient Credential Hygiene:

Although we did able to perform PPE successfully it was not necessarily possible to print the credential the in the console and that lead another misconfiguration i.e. Insufficient Credential Hygiene

ICH.png

It might be the secret(flag) is stored in the Jenkins credential store with the Global scope, which makes it accessible to any pipeline on the Jenkins instance.

Mitigation

ICH-Mitigations.png

As in this particular case its been set to Global Scope - refer Jenkins credentials stored with global scope

Mad Hatter

chl2_desc.png

Hint 1: Where’s the Jenkinsfile stored? Search for the repo name in the Wonderland organization, you might find some helpful repos.
Hint 2: What commands are run by the Jenkinsfile?

Looking for Jenkinfile in mad-hatter repo is visible no visible that reminded me of Hint1 tha says Jenkinfile is stored in somewhere else, did the same and found it

mad-hatter-search.png

that shows real life example that you may come across that its been stored sperate from repository for security purpose, lets read the file.

Jenkinfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
pipeline {
    agent any
    environment {
        PROJECT = "yagmail"
    }
    stages {
        stage ('Install_Requirements') {
            steps {
                sh """
                    virtualenv venv
                    pip3 install -r requirements.txt || true
                """
            }
        }
        stage ('Lint') {
            steps {
                sh "pylint ${PROJECT} || true"
            }
        }
        stage ('Unit Tests') {
            steps {
                sh "pytest || true"
            }
        }
        stage('make'){
            steps {
              withCredentials([usernamePassword(credentialsId: 'flag3', usernameVariable: 'USERNAME', passwordVariable: 'FLAG')]) {
                sh 'make || true'
              }
            }           
        }
    }
    post { 
        always { 
            cleanWs()
        }
    }
}

Make stage seems to interesting lets look at it

1
2
3
4
5
6
7
stage('make'){
    steps {
      withCredentials([usernamePassword(credentialsId: 'flag3', usernameVariable: 'USERNAME', passwordVariable: 'FLAG')]) {
        sh 'make || true'
      }
    }           
}

its using Credential Binding technique that allows credentials to be bound to environment variables for use from miscellaneous build steps.

and sh 'make || true' command that cause the pipeline to ignore any errors that occur during the make step. This means that even if the make command fails, the pipeline will continue running.

lets the repeat the same technique to print the flag to the console and if see if it works or not, so i created new branch and make changes to jenkinfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
pipeline {
    agent any
    environment {
        flag3 = credential("flag3")
    }

        stage('Printing the flag'){
            steps {
              withCredentials([usernamePassword(credentialsId: 'flag3', usernameVariable: 'USERNAME', passwordVariable: 'FLAG')]) {
                sh 'echo $FLAG | base64'

              }
            }           
        }
    }

    post { 
        always { 
            cleanWs()
        }
    }
}

push_failed1.png

but this time we have no luck, as this time it has implemented branch security even not allowed to push the code.

trying with main branch gives us the same error

push_failed2.png

and this made us to perform [I-PPE] Poisonig attack by trigerring the pipeline indirectly without modifying the Jenkinfile

As we saw that Jenkinfile taking use of Makefile that means it runs the all the command that Makefile contain so this time we will gonna poison the Make file rather than Jenkin one, where we can simply print the Environment Variable that Credentials Binding plugin bind.

so i created new branch and committed the makefile there

makefile.png

push_success.png

And it worked, so the pipeline was triggered and we can get the flag in the console now

chl2_flag.gif

Learning Takeaway

We leveraged the misconfiguration of one of PPE attack i.e. I-PPE(Indeirect Pipeline Posion Execution)

I-PPE.png

Mitigation

  1. Branch Protection Rules:
  • Implement branch protection rules in your Source Code Management (SCM) system (e.g., Git).
  • Enforce mandatory code reviews for any changes to the Jenkinfile or Makefile before merging them into a branch that triggers the pipeline.
  • Restrict who can push directly to branches that trigger pipelines. Consider requiring merges from a protected branch reviewed by a trusted developer.
  1. Separate Configuration Files:
  • Store the Makefile in a separate repository that’s not publicly accessible or easily modifiable.
  • Reference the Makefile location within the Jenkinfile using a secure mechanism like environment variables or secrets management tools.

Here developer could have follow Proper Credential Hanlding technique e.g. Using withCredentials blocks and ensure sensitive information is not printed. Use Jenkins’ masking feature to hide sensitive information.

1
2
3
4
5
6
7
withCredentials([usernamePassword(credentialsId: 'flag3', usernameVariable: 'USERNAME', passwordVariable: 'FLAG')]) {
    sh '''
        set +x  # Disable command echoing
        make || true
        set -x  # Enable command echoing
    '''
}

and Limit access to the Jenkins environment and critical files to only those who absolutely need it.

Duchess

Description

duchess_desc.png

Hint1 : A PyPi token has a prefix of “pypi-”.
Hint2 : Mistakes could have been made in the past.

Well did you hear this quote ever?

Once a secret is committed to Git, it can never be considered secret again

if yes you got the challenge solution or else lets see what this challege is about

Reading description its clearify that developer/s made some mistake that they leftover any credential i.e. token i.e. PyPi token in repository that may present in SCM history i.e. commit history if proper/any mitigation has not been applied, So our job is to find the token that is most likely the flag?

hoping into repository we can see that there are 696 commits

commits.png

we can look for that particular commit that contain the token manually by checking it one by one that is obviously tedious or we can use keyword that Hint1 gave us

duchess_flag.gif

and here we go we got the flag!!

BTW there is more efficient way to search for leaked credentials that is by using automated tools like - TruffleHog, GitGuardian, or Gitleaks.

Here we are going to use Gitleaks, lets see how to get the flag by using it

gitleaks.png

Learning Takeaway

# CICD-SEC-7: Insecure System Configuration

As i said earlier Once a secret is committed to Git, it can never be considered secret again this particular misconfiguration falls into Insecure System Configuration type that says

Insecure-sys-config.png

Exposing sensitive information such as tokens, passwords, API keys, or other confidential data in the version control history.

Mitigations

To mitigate this type of vulnerability, you can take several steps:

Secrets Management:
- Use secret management tools to handle sensitive information. Tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault can help securely store and manage secrets.

Pre-commit Hooks:
- Implement pre-commit hooks to prevent committing sensitive data. Tools like git-secrets or pre-commit can be configured to scan for sensitive data before commits are made.

Scanning for Secrets:
- Regularly scan your repository for exposed secrets using tools like TruffleHog, GitGuardian, or Gitleaks. These tools can detect and alert you to the presence of sensitive information in your repository.

Removing Secrets from History:
- If secrets have already been committed, remove them from the repository history using tools like git filter-branch or BFG Repo-Cleaner. This process rewrites the commit history to remove the sensitive data.

Rotating Credentials:
- If a secret has been exposed, immediately rotate the affected credentials. This involves generating new tokens or passwords and updating any systems that use the old credentials.

Educating Developers:
- Train developers on the importance of not committing sensitive information to version control and on best practices for managing secrets.

Monitoring and Alerts:
- Set up monitoring and alerting to detect any unauthorized access or misuse of credentials that might have been exposed.

Or refer these links:
- https://github.com/DEFRA/software-development-standards/blob/master/processes/credential_exposure.md
- https://blog.gitguardian.com/leaking-secrets-on-github-what-to-do/

Part 2 - here