Files deleted or modified between git versions are automatically deleted from instances

Question

Background

I have a setup initiated by Jenkins with the following:

• The files to be deployed are prepared by phing, talking to the git server and accepting git diff between the required git versions on a separate build server without involving AWS code deployment (as far as I think). The phing line is started by Jenkins.
• I add only the files that need to be added / modified (based on the difference of git reverses) dynamically in the appspec.yml file. I only prepare files for adding / changing to the path /home/jenkins/deployment/cd_deploy/codebase/ , and I specified the path /home/jenkins/deployment/cd_deploy/ in the "Use custom workspace" section in the "Advanced project settings" section the Jenkins project, which is basically the location on the build server that needs to be loaded into the S3 bucket. Note that I will need to remove files from instances deleted between the two versions of git.
• Jenkins then launches AWS Codedeploy with information about the name of the application, the code deployment deployment group that I configured.

Problem

The files that I dynamically add to the appspec.yml file are modified / added to EC2 instances, as I expect, however, it is strange that the files that should be deleted are also deleted. I confirmed that I have no logic to delete those files that were recorded in the beforeInstall file for my appspec file. I have only the beforeInstall.sh file in my beforeInstall cache, and there is no other hook. As soon as I remove this hook from the appspec file, the removal will stop. Here is my application file -

 version: 0.0 os: linux files: {Pair of files dynamically generated} - source: config/deployment_config.json destination: /var/cake_1.2.0.6311-beta/deployment permissions: - object: . pattern: "**" owner: sandeepan group: sandeepan mode: 777 type: - file hooks: BeforeInstall: - location: beforeInstall.sh 

Is AWS Codedeploy somehow talking to my git host (I use gitlab and even github) and somehow get information about the files that need to be deleted.

Update

Later, I noticed that even after completely removing the hooks section from the appspec.yml file and deleting the corresponding .sh files, i.e. beforeInstall.sh, afterInstall.sh, etc. from the central build server (where the S3 package is ready), so that none of my logic and no link to it goes to the instances, the files that should be deleted are automatically deleted anyway.

Update 2

Today I discovered that files that have been changed between versions of git are also automatically deleted. I had the logic for dynamically preparing the appspec.yml file. I changed so as not to add some files. So there were some files that were there in git diff but not in the appspec file. As a result, they are deleted, but do not appear again. Deploying code automatically performs a cleanup before deployment. How to stop it? I would like to add my own cleanup logic.

Update 3

Contents beforeInstall.sh -

 OUTPUT="$(w | grep -Po '(?<=load average: )[^,]*')" rm -f /var/cake_1.2.0.6311-beta/deployment/deployment_config.json path="$PWD" php $path"/deployment-root/"$DEPLOYMENT_GROUP_ID"/"$DEPLOYMENT_ID"/deployment-archive/beforeInstall.php" ${OUTPUT} /usr/local/nagios/libexec/check_logwarn -d /tmp/logwarn_hiphop_error /mnt/log/hiphop/error_`(date +'%Y%m%d')`.log #Just run a nagios check, so that counter corresponds to the line in the log corresponding to current timestamp/instant. Do not care about output. Note that we are not even looking for error hinting keywords (and hence not using -p because it needs to be used alongwith), because all we need to care about here is incrementing the nginx counter. /usr/local/nagios/libexec/check_logwarn -d /tmp/logwarn_nginx_access /mnt/log/nginx/access_`(date +'%Y%m%d')`_`(date +'%H')`.log #Just run a nagios check, so that counter corresponds to the line in the log corresponding to current timestamp/instant. Acceptable http codes are also not being read from deployment_config.json. printf "\n `date +%Y-%m-%d:%H:%M:%S` End of beforeInstall.sh" >> /var/cake_1.2.0.6311-beta/deployment/deployment.log exit 0 

And the contents of beforeInstall.php, which is called from the above -

 <?php file_put_contents('/var/cake_1.2.0.6311-beta/deployment/deployment.log', "\n ".date("Ymd H:i:s")." - Load print ".$argv[1], FILE_APPEND); $loadData = json_encode(array("load" => intval($argv[1]), "access_error_check_day" => date("Ymd"), "access_error_check_hour" => date("H"))); //error_check_day -> day when nagios error check was last run. We will accordingly check log files of days in between this day and the day of afterinstall (practically this can include a span of 2 days). file_put_contents("/var/cake_1.2.0.6311-beta/deployment/serverLoad.json",$loadData); //separate from deployment_config.json. serverLoad.json is not copied from build server. file_put_contents('/var/cake_1.2.0.6311-beta/deployment/deployment.log', "\n ".date("Ymd H:i:s")." loadData to config ".$loadData, FILE_APPEND); ?> 

Answer 1

CodeDeploy is designed to deploy applications, not just copy a specific and constantly different set of files.

Thus, before deploying each “revision”, CodeDeploy will first clear all files deployed by the previous version. Let me explain.

So, let's say the previous application deployment uploaded three files:

 File A File B File C 

And then the following deployment included only these files:

 File A File C 

Code Deploy will first clear the 3 files that it deployed in the first revision (A, B and C), and then deploy the new version ... It never just downloads the intended files, it always cleans up the old files first (determined by looking at the previous " editors "). This is important because it sheds some light on what seems like mysterious behavior in your case. The result, after deployment, of course:

 File A File C 

Now I wonder if you manually added files to the composition outside of CodeDeploy. It will clean only those things that it knows about, and will also not overwrite files in the current version if this cleaning phase does not delete them. This is often observed when people manually installed the application, and then tried to make CodeDeploy in the same folder ... there is no previous revision, so nothing needs to be cleared, and then it tries to copy over existing files and an error will occur. Usually you want your target folder to be bare so that you can run the change history correctly.

For example, in the previous scenario, if you preloaded the A, B, and C files manually, the deployment of the A and B files would fail, because it was not known to clear A, B, and C first, and then you get a message about error to overwrite files A and B.

A file (or folder) is completely beyond the scope of deployment ... i.e. is not part of any revision, say file D ... will not be affected and will remain happy there before and after deployment without complaint. This is useful for hosting data files and things that may be deployment specific, but are not necessarily part of a code base that you don’t want to constantly relocate.

Now you can do a lot of interesting things using hooks, of course, but this seems like the wrong tool to work with. Hooks are designed to perform functions such as stopping / starting a service, etc., so as not to control file copy control, which is the cornerstone of what CodeDeploy should do for you.

Excluding all files from the application specification (i.e., files not specified) and simply using the BeforeInstall and / or AfterInstall steps to execute the copy logic is an approach that may work for some scenarios.

In any case, perhaps this is a better understanding of how CodeDeploy works, can help you develop a solution. I do not think that he was especially well documented. My understanding comes from observing and dealing with it.