2 posts tagged with "jenkins"

I would like to share a very simple method for optimizing resource costs when dealing with batch applications that need to run at specific times and under specific conditions.

Problem​

1. Batches are only executed at specific times. For tasks like calculations, which need to run at regular intervals like daily, monthly, or yearly.
2. Speed of response is not crucial; ensuring that the batch runs is the priority.
3. Maintaining an EC2 instance for 24 hours just for resources needed at specific times is inefficient.
4. Is it possible to have the EC2 instance ready only when the cloud server resources are needed?

Of course, it is possible. While there are various automation solutions like AWS ECS and AWS EKS, let's assume managing batches and EC2 servers directly with Jenkins and set up the environment.

Architecture​

With this infrastructure design, you can ensure that costs are incurred only when resources are needed for batch execution.

Jenkins​

Jenkins Node Management Policy​

Activates the node only when there are requests waiting in the queue, minimizing unnecessary error logs. Additionally, it transitions to idle state if there is no activity for 1 minute.

AWS CLI​

Installing AWS CLI​

With AWS CLI, you can manage AWS resources in a terminal environment. Use the following command to retrieve a list of currently running instances:

aws ec2 describe-instances

Once you have checked the information for the desired resource, you can specify the target and execute a specific action. The commands are as follows:

EC2 start​

aws ec2 start-instances --instance-ids {instanceId}

EC2 stop​

aws ec2 stop-instances --instance-ids {instanceId}

Scheduling​

By writing a cron expression for the batch to run once a month, you can set it up easily.

H 9 1 * *

Now, the EC2 instance will remain in a stopped state most of the time and will be activated by Jenkins once a month to process the batch.

Conclusion​

Keeping an EC2 instance in a running state when not in use is inefficient in terms of cost. This article has shown that with Jenkins and simple commands, you can use EC2 only when needed.

While higher-level cloud orchestration tools like EKS can elegantly solve such issues, sometimes a simple approach can be the most efficient. I hope you choose the method that suits your situation best as I conclude this article.

Overview​

This article explains how to install and operate Jenkins using Docker.

Contents​

Install​

Docker​

docker run --name jenkins-docker -d -p 8080:8080 -p 50000:50000 -v /home/jenkins:/var/jenkins_home -u root jenkins/jenkins:lts 

Mount a volume to persist Jenkins data on the host machine. Unlike TeamCity, Jenkins manages all configurations in files. Setting up a mount makes authentication information and data management much more convenient, so be sure to configure it. Common target paths are /home/jenkins or /var/lib/jenkins.

For the purpose of this article, it is assumed that the path /home/jenkins has been created.

Authentication​

To ensure security and access control for both the master and nodes, create a user named 'jenkins' and proceed as follows.

Setting User Access Permissions​

chown -R jenkins /var/lib/jenkins

Managing SSH Keys​

If you don't have keys, generate one using ssh-keygen to prepare a private key and a public key.

When prompted for a path, enter /home/jenkins/.ssh/id_rsa to ensure the key is created under /home/jenkins/.ssh.

GitLab​

In GitLab's personal settings, there is an SSH setting tab. Add the public key.

When selecting Git in the pipeline, a repository path input field is displayed. Entering an SSH path starting with git@~ will show a red error. To resolve this, create a credential. Choose SSH credential to create one, and the ID value can be a useful value, so it is recommended to enter it.

Node Configuration​

Nodes are a way to efficiently distribute Jenkins roles.

To communicate with the node, generate a key on the master using ssh-keygen. If you already have one that you are using, you can reuse it.

• ID: This value allows Jenkins to identify the SSH key internally, making it easier to use credentials in Jenkinsfiles, so it's best to set a meaningful value. If not set, a UUID value will be generated.
• Username: The Linux user. Typically, 'jenkins' is used as the user, so enter 'jenkins'. Be cautious as not entering this may result in a reject key error.

Docker Access Permissions​

If the docker group does not exist, create it. Usually, it is automatically created when installing Docker.

sudo groupadd docker

Grant Jenkins user permission to run Docker by running the following command.

sudo gpasswd -a jenkins docker
# Adding user jenkins to group docker

sudo chmod 666 /var/run/docker.sock

Restart the Docker daemon to apply the changes.

systemctl restart docker

You should now be able to run the docker ps command.

Restart​

When updating Jenkins version or installing, removing, or updating plugins, Jenkins restarts. However, if you are managing it with Docker, the container goes down, preventing Jenkins from starting. To enable restart, you need to set a restart policy on the container.

docker update --restart=always jenkins-docker

After this, the jenkins-docker container will always remain in a running state.

Caution​

When updating plugins, carefully check if they are compatible with the current version of Jenkins in operation. Mismatched versions between Jenkins and plugins can often lead to pipeline failures.

Reference​

Managing Jenkins with Docker