Recently I was working with a customer who had a number of IBM Datastage jobs that all had set of standard schedules and varying unique schedules. These all needed to be dependent on each other. When trying to solution this, I ran into an issue trying to incorporate all of the job flows into one big flow. This was because all of the unique schedule criteria made things difficult. For example we had easy jobs that ran on M-F, but we needed to incorporate the jobs with the unique job schedules into the jobs with the easy schedules. Lets say that JOBA, JOBB and JOBD are the M-F jobs. JOBTHURSC is only scheduled to run on Thursday. Without using the “CTM_FOLDER_ADJUST_DUMMY” parameter the jobs would order M-F but the JOBTHURSC would not be present except on Thursdays. This will cause you issues everyday except for Thursday because JOBB would finish and not see JOBTHURSC and would cause JOBD never to execute because it would be looking for JOBTHURSC.
To solve for this I came across the “CTM_FOLDER_ADJUST_DUMMY” parameter. By adding the parameter into my environment ,this allowed me to solve for my Thursday job always being in the flow as a DUMMY JOB but only running when it was not its day to run. You will then have to create a Smart Folder for your jobs. The parameter tells the Smart Folder to always order every job in the Smart Folder according to the scheduling criteria of the Smart Folder. If a job’s individual schedule is not that particular order date then set it to run as a “DUMMY” job. If you look closely at the job flow screenshot,you notice the purple ghost symbol. (I think it is the bad guys from Pac-Man.) This is the indicator that the job is set to dummy. When I tested this flow out it was a Monday, so all the jobs ordered in the Smart Folder because in this case the Smart Folder’s schedule is set to EVERYDAY. The Thursday job ordered even though it was Monday but it ran as a DUMMY JOB. So the DUMMY JOB in this case called JOBTHURSC becomes a placeholder to keep the flow inline without having any gaps because it is not Thursday. This solution very happy. The customer’s current setup within Datastage Director did not account for this and caused them to have to run some jobs manually at particular times to accommodate for their needs. Meaning somebody from that team might be getting up at 5am to kick off things in certain time windows. Hopefully this solution keeps allows the customer to focus more on their other Datastage tasks and leave the scheduling to me and Control-M.
Listed below are the instructions for Version 8 from the Knowledgebase.
1. Log in to the Control-M Configuration Manager.
2. Right click on the Control-M/Server that this table resides on and choose system parameters.
3. Find the parameter named “CTM_FOLDER_ADJUST_DUMMY” and change the value to Y.
4. Restart the Control-M/Server
5. Go back to the planning environment and create a Smart Folder for the jobs you need to solution this for.
6. In the Smartfolder properties select the pre-requisites tab. Within there will be an option called Adjust Condition. Set this to YES.
Last night I had the very annoying happening when I was building some test jobs in Control-M. The test jobs were pointed to an agent where the jobs would be executing the scripts directly on the Control-m agent. I ordered the jobs into the Monitoring environment and let them to run in the order that I created them in their workflow. Job 1 was only supposed to run for 1 minute. It ended up running for 5 minutes. I immediately knew something was wrong. When I attempted to read job 1’s output I was unable to retrieve this from the job. I also tried to kill the job. This action was rejected by Control-M. I started to panic because I had never seen this before. I had job 1 set to run all day so I changed it to stop running in one minute. It stopped running and failed. I extended the time a few minutes and tried to re-run it again. It resulted in the same issue as before. This time I killed the services on the agent and rebooted it. This sent the job into a unknown state. That didn’t help.
I then decided to go to the agent that was to execute this job and went to the following path \Program Files\BMC Software\Control-M Agent\Default\PROCLOG\. In the PROCLOG folder I started looking the logs that started with AM_SUBMIT*. And look for logs that were created around the same time my job was created. I also searched within the AM_SUBMIT* log with my the job ID given to the job when it was ordered into the Control-M Monitoring environment. (The job ID is in the Synopsis area for the job) This reassured me I was in the correct log. I scanned for errors and found the following ,“CreateProcessAsUser failed with error: 1314 – A required privilege is not held by the client”, and immediately went to the BMC website and went to the Support Central area. I then started looking through the Knowledge Articles for this errors. Luckily I wasn’t the only Control-M admin to run across this issue.
To resolve this problem, you need to increase the rights of the job owner, adding the “Replace a process level token” permission. To do so, please follow the following instructions:
1) Open the Control Panel / Administrative Tools / Local Security Policy
2) Add to the job owner user account the “Replace a process level token” right. (You may have to logout or even reboot to have this change take effect.)
If another user account other than the “Local system account” is defined under the “log on” tab of the Control-M/Agent service, then the account should be member of the local admin group and have the following rights:
Local Security Settings:
> Adjust memory quotas for a process
> Replace a process level token
> Log on as a service
List Folder Contents permissions for the agent drive
I was fortunate because all I needed to was add the “Replace a process level token” right for my control-m account that was executing the jobs on that server. Then I rebooted. After I did this I immediately re-ran my job and it worked properly and returned a output. I proceeded to execute the other test jobs and they all worked correctly. Problem solved.
First proceed to your Configuration Manager and right click on the Control-M Cloud module and go to Connection Profile Management. Then click the + for a new profile. Give your profile a name then click the dropdown to the VMware selection.
Now in the account creation page you will notice several fields that you need to fill in. This information will be provided by your VMware VCenter application owner. The fields include the Web Services Server Host Name, Web Services Server Port, Service Path. These fields will build out the Server URL that the Control-M for Cloud module/agent it will use to communicate with VCenter. Also you can use the check marks to turn SSL on or off, use the default listening port for VCenter, and to use the Default Service Path. If you need to use SSL reach out to BMC Support for assistance for how to import the SSL configuration file into the agent server the Control-M for Cloud module resides on. Also there is information related to importing the SSL configuration file on the BMC website. If you aren’t required to use SSL just do not check “Use SSL”. The last three fields will include the Username and password that Control-M for Cloud will use to access VCenter.
In my shop we used an existing control-m system account and password. The VCenter admin gave it the appropriate rights and access to be used in VCenter. In some cases the VMware admin can limit which three categories (Power Tasks, Snapshot Tasks, Configuration Tasks) of Control-M for Cloud the system account will have rights to execute. Also another thing I wanted to point out is in the beginning of our testing phase the admin only granted me rights to execute Power Tasks. As things start moving along the system account will eventually have rights to execute Snapshot operations as well. This is fine because it all depends on what your IT shop needs you to accomplish for them. At this time our shop just needs to automate reboots at odd hours. Eventually we hope to be involved with executing snapshots as well.
Once you have all the information you need click test and if everything authenticates without issue you will see a popup windows with a green check mark letting you know the connection is good. If you do not get this green check mark then you will have to continue to work with your VMware admin to work through the connection issues. Just remember it’s going to be something simple such as firewall issues, port issues, or you just fat fingered something in the account information field. So keep it simple when troubleshooting your Control-M for Cloud account setup issues. If you are dealing with SSL you might run into some issues such as incorrectly import the SSL file by making a mistake in your import script or your VMware admin might have give you the wrong SSL file to import. I recommend if you get stuck just turn your logging on and pipe out the proc logs and open a ticket with BMC Support to help you narrow down your issues. I recommend doing this anytime you get stuck with any type of setup for a Control-M module. BMC Support will help you get through any type of issue.
Recently I was working on documentation for my production environment. I was clicking through some of the folders and looking at jobs. I came across a few jobs that were decommissioned a few months back. I went to delete these out of the folder in planning section. In the process of highlighting multiple jobs, I accidentally deleted one of my production jobs in the process. This led me to learn how to recover a deleted job(s) from my planning production environment. In fact I had a good colleague that had the same incident happen to him and he had to work magic in his Control-M database to recover his jobs. Hope these directions below can make it simple for those who have made the same mistake as I have.
When discovering you deleted a production job. Go to the planning selection and then click on the Tools icon. Then click on Versions. After clicking on the Versions icon click on the correct Control-M you are using then select the folder that the job was deleted from. Then use a previous days date before the jobs were deleted. Then click on the Deleted Items from the Change Type: menu. After this select Restore in the bottom right.
Notice the job has the word Deleted in the Change Type field. Make sure this is the job you were missing and then click restore. This will bring a the job back in alone in a folder that was named the folder as it was in before it was deleted. Make sure you copy the job once it is restored. Get rid of the folder that it restored in then go back to the folder it was missing from and paste it back into the folder where it was originally Then check this change back in and now you will have your job back.
I recently attended a first time Control-M User Group conference in the Atlanta, Georgia area. It was organized by a fellow Georgian and my favorite BMC rep. He did an outstanding job of putting on the event. It had a very nice turnout and some of the BMC technical staff showed up as well. Even better, it’s also nice to know that there are other fellow Control-M folks close by in my area. We discussed the latest Workload Automation Change Tool and a product called Control-M for Hadoop. We also covered some basics of Control-M Version 8 for a lot of the audience member’s whose environments were still running previous versions of Control-M. Of course we had good food and SWAG too!
This first meeting has set a good foundation to possibly make an official User Group at the next event. I think the frequency of these meetings should be quarterly. This will give us plenty of time to arrange to be out of the office and keep the event from getting watered down. As things move forward I would really like to see our User Group stay consistent. We need to keep these events organized and open for good discussions. In the future our members do not need to be afraid to ask questions, share horror stories, share awesome stories and those Aha moments! Every time we attend these events we need to remember to make our rounds and introduce ourselves to others. Maybe we could arrange games to allow all attendees to introduce themselves. Maybe a little Control-M trivia would be cool? Such as what was the Hadoop product named after? Also these User Group Events need to have a lot of SWAG from BMC. I mean we had pretty good swag at this User Group meeting but I think we should have the good stuff every time. Also lets make sure we always have good food.(The food was good at this one) I am just saying this group will not be allowed to do pizza. Our User Group needs to be able to listen to it’s members and provide answers that they think might help. Also if technical BMC Control-M experts are in attendance we can leverage with them on Control-M topics as well. These are just a few of my ideas. Looking forward to hearing more ideas from others! Regardless of what is decided, the end result should be I can’t wait to try that in the office tomorrow!
I recently added Control-M for Cloud module to my Control-M Version 8 environment. It is a very easy install just like any other Control-M module. I remember hearing about it in 2013 at the Control-M Briefing conference in Houston at BMC Software headquarters. During the Control-M for Cloud presentation I was very impressed with the information the presenter provided for all of the Control-M professionals with in the audience. The presenter touched on how Control-M for Cloud can spin up servers using Amazon Elastic Compute Cloud web services (Amazon EC2), how it works with VMWare VCenter to automate all of the types of tasks in that application from reboot, snapshots and migrations, and also how it works with BMC’s product Bladelogic which does server automation related tasks. At the time of the presentation, I knew that I wouldn’t be using much of Control-M for Cloud but I did see some possible potential for this module in the future. Upon arriving home from the briefing I made sure to pass the information about Control-M for Cloud to one of my IT shop’s senior system engineers.
Now flash forward from February 2013 to March 2014. The same Senior who I shared the Control-M information with happened to email my manager about Control-M for Cloud and it’s capabilities. We had a customer who realized that they was having issues with their application crashing when it’s Linux servers were not rebooted at least once a week. The issue was they weren’t very well versed in Linux to navigate around and to do reboots. So to solve this we decided to use Control-M for Cloud to automate the reboot task and send them a confirmation via email that the reboot took place for them instead of having our Linux expert write a script to do this or have the Linux expert handle the reboots every Sunday at 5:00am.
The next step involved me gathering more information on the product. I used the PDFs off of the website and also reached out to colleagues. I even looked at Webinars and Youtube videos that BMC pushed out for its followers to reference for help. After reading a good bit on the product, I went on the the BMC website signed in with my account and went to the Product and Downloads and Patches area on BMC’s website and found what I was looking for. Then used the Electronic Product Distribution option to get the correct version and platform of Control-M for Cloud and then I had the zip file for the install sent to me via email all by using EPD. Once I had the zipfile on my desktop, I copied it on to the Control-m agent server, where I intended the Control-M for Cloud module to run. I extracted the zip file info and then once it finished extracting went on to click on the setup file that was populated from the zipfile extraction. This brought up an BMC install window that you walk through during the install. All you really need to know is the name of the Control-M agent server you are installing Ctrlm for Cloud on and the host server where the Enterprise Manager is located. It’s just clicking next several times and then it will begin to download. Once it completes click finish. Then you will need to go log into your host or main Control-M apication server where the enterprise manager is located and run some commands from the command line. Once these commands are successfully executed your install of Control-M for Cloud is completed. (And these commands are what you run after any Control-M module install.)
Here are the commands you need to use to complete your module install in parenthesis below.
(After completion of your Control-M module go to the main application server. Sign in as the Admin Control-M account. Log on to CMD prompt and type in “ctmgetcm”. Then enter Which is the agent server name you just installed the AFT module on to. Press “Enter”. Enter : [*]: Enter “*” then press “Enter”. Enter : [View]: Enter “Get” then press “Enter”. )
Alright now once this is completed your Control-M for cloud installation is officially completed.
On the next blog I will show you how to define a Control-M for Cloud VMWare Account.
Hey Control-m users and followers. I would like to discuss how to handle control-m system accounts on Windows servers that have unexpected or planned domain changes on Windows servers that are a part of jobs in your distributed production control-m job environment.
Recently in my environment we have had a number of domain changes which have caused a number of my jobs to fail. This primarily being because the changes were not communicated all the way across the board.
To solve this issue for Control-m I had to troubleshoot a few of the problematic jobs. This led me to realize what the issue was. Our traditional domain name was slowly being phased out on a few servers at a time. It was making the jobs fail because the traditional domain control-m system account was not of any use in the new domain. I knew I had a new domain control-m system account created just did not have to use it yet. To fix this I placed a request to the Active Directory admin to add the new domain control-m system account to the servers involved with my failed jobs that had the domain change, and also to give it read, write and delete rights. Once the admin completed my request, I also added the new system account in the actual job itself where it says runas. Then I re-ran all my jobs that failed because of this reason and they began working again.
To be proactive I plan on adding the the new domain control-m system account to all of the servers my jobs touch that still have the traditional domain name. That way as they’re migrated to the new domain the domain system account will be on the server. That way access and rights will not be effected and none of my customers will be impacted.