Sometimes, when you start in new organisation, there is another dedicated team who creates AWS account for you and your role is restricted to manage AWS resources either via portal or CLI.
You try to automate the things and then as part of this automation you setup AWS CLI
There is one strange situation where, you are able to create/manage/destroy resources from the AWS Web Console but when you try to do the same through CLI – you are getting “AccessDenied”, “UnauthorizedOperation” and “You are not authorized to perform this operation” errors for all sort of actions, such as:
[root@automation-vm]# aws ec2 describe-security-groups
An error occurred (UnauthorizedOperation) when calling the DescribeSecurityGroups operation: You are not authorized to perform this operation.
[root@automation-vm]# aws ec2 describe-vpcs
An error occurred (UnauthorizedOperation) when calling the DescribeVpcs operation: You are not authorized to perform this operation.
[automation-vm]# aws ec2 describe-instances
An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.Lets try to fix this :-
The very first thing you check is your Access keys with aws CLI
[root@automation-vm]# aws configure list
Name Value Type Location
---- ----- ---- --------
profile <not set> None None
access_key ****************7ZGO shared-credentials-file
secret_key ****************9ft6 shared-credentials-file
region eu-west-1 config-file ~/.aws/config
Its perfectly fine as we didn’t get any authentication error like
An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentialsThe next step is to verify IAM Policies and Permissions.
This also looks fine. Lets try to run the command to create the AWS instance and see the outcome
[root@automation-vm]# aws ec2 run-instances --image-id ami-0ce1e3f77cd41957e --count 1 --instance-type t2.micro
An error occurred (UnauthorizedOperation) when calling the RunInstances operation: You are not authorized to perform this operation. Encoded authorization failure message: U9CRdJngzzYdO3DLoYRcjxlVqLcxtZ9D74GYS61GRdFbyasjEME4VDH9TqAwzzmTjkt_2v5HuGl90zw3XT7l_IEKedG2cR6MKUNy7N0-vj1YsdCk9wl5Lf0VyctWbzghsNWUNfBU5X2izecx3nJeGjJCygOCGTHklKrpxq8Ytex_WRHnOkiScj40KC0B46l9k0z2CtHRPzkcKC89Gp9JyDMuqpJ-LhyPzfA-hM1gIyloaXRkcH6t49_VNqssWht3vKW9zpLpnpNWILINWi7u4fgz8g4M3E1W1MHDLX365nA-Z3rmVl5f4zG6qLZ3Vsq9HzYhp5rjUJla-usjBzlxrUrlhlps2S4-OGSAdFUGClqAjAFAkdfBr4n0albOL1v7sZ0p6BYgPanySEl36tSY_uLpowzQeAhXk3dyktT-F-FBCn7GDj5dq39Wb0LsZP7pV5Qt0PfkMFaC9XN148RSfU18tX-xyO6sShQoOu0Cv2BKy2HpyO_-NNrG208Pk6lvMsAFDeoZfCbm1R8Bt4ZTjSbhtkyzi-obQR086VNbspY5uwkY7mqyE-eAqh6j8tc0GjOkK0XtYajh04EatHcceeNAAMj5rtdH7yRxq6uTdzsVxJ0ZTav10NJwpkWWFQTY5Y1XrBGhjSRPFdNez1CzZ4DZOupz13DKGnXZO-pX0ZKTEEJEA6c1fehwhFWAFwoDOuPB4xgP8Jy0ETd63ERGxBv7qdAklpynrEEwp970dOnIvy7g5VB7M87oboBbyVqMc24hDrs6hAXDkS7lWxCw6oxq99OG4Vms2w1zUM5YUJYGWT82BsaFrP2QJGHZcHw5jfu1A0Ow4y0IoQviB5wB0A2c_vEAxkLche1oMkw87sy1YZDiaW4glj_6l7ZVSnW2Y6iF0PW8Qd3fezuCgV3LjY4TPJyR1OD1rL05TGocEHKJ
Now, we got some different encoded error which needs to be decoded by sts
You may need to apply the inline policy to the user in order to run the sts execution
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowStsDecode",
"Effect": "Allow",
"Action": "sts:DecodeAuthorizationMessage",
"Resource": "*"
}
]
}Let’s try to decode it after applying the policy.
[root@automation-vm]# aws sts decode-authorization-message --encoded-message U9CRdJngzzYdO3DLoYRcjxlVqLcxtZ9D74GYS61GRdFbyasjEME4VDH9TqAwzzmTjkt_2v5HuGl90zw3XT7l_IEKedG2cR6MKUNy7N0-vj1YsdCk9wl5Lf0VyctWbzghsNWUNfBU5X2izecx3nJeGjJCygOCGTHklKrpxq8Ytex_WRHnOkiScj40KC0B46l9k0z2CtHRPzkcKC89Gp9JyDMuqpJ-LhyPzfA-hM1gIyloaXRkcH6t49_VNqssWht3vKW9zpLpnpNWILINWi7u4fgz8g4M3E1W1MHDLX365nA-Z3rmVl5f4zG6qLZ3Vsq9HzYhp5rjUJla-usjBzlxrUrlhlps2S4-OGSAdFUGClqAjAFAkdfBr4n0albOL1v7sZ0p6BYgPanySEl36tSY_uLpowzQeAhXk3dyktT-F-FBCn7GDj5dq39Wb0LsZP7pV5Qt0PfkMFaC9XN148RSfU18tXxyO6sShQoOu0Cv2BKy2HpyOpX0ZKTEEJEA6c1fehwhFWAFwoDOuPB4xgP8Jy0ETd63ERGxBv7qdAklpynrEEwp970dOnIvy7g5VB7M87oboBbyVqMc24hDrs6hAXDkS7lWxCw6oxq99OG4Vms2w1zUM5YUJYGWT82BsaFrP2QJGHZcHw5jfu1A0Ow4y0IoQviB5wB0A2c_vEAxkLche1oMkw87sy1YZDiaW4glj_6l7ZVSnW2Y6iF0PW8Qd3fezuCgV3LjY4TPJyR1OD1rL05TGocEHKJ
An error occurred (AccessDenied) when calling the DecodeAuthorizationMessage operation: User: arn:aws:iam::57*********9:user/toolsadmin is not authorized to perform: sts:DecodeAuthorizationMessage with an explicit denyBut even this is not working – What to do now ?
One of the simplest answer is : User policies blocks the CLI access due to MFA
Review the IAM Policy attached to the user which refers to Multi Factor Authentication
{
"Sid": "DenyAllExceptListedIfNoMFA",
"Effect": "Deny",
"NotAction": [
"iam:CreateVirtualMFADevice",
"iam:EnableMFADevice",
"iam:GetUser",
"iam:ListMFADevices",
"iam:ListVirtualMFADevices",
"iam:ResyncMFADevice",
"sts:GetSessionToken",
"iam:ChangePassword",
"iam:GetAccountPasswordPolicy"
],
"Resource": "*",
"Condition": {
"BoolIfExists": {
"aws:MultiFactorAuthPresent": "false"
}
}
}Did you notice, here admin enforces MFA authentication for IAM users that use the AWS CLI.
It could be one of the requirement for the team who manages AWS access, so you have few choices :
- Request them a new account without MFA and just for Programmatic access
- Detach the policy from your user account which requires lot of approvals and discussion
- Check with the team if they have explicitly put this option, if they haven’t – then simply follow the below solution
Change the BoolIfExists condition to Bool Condition
{
"Sid": "DenyAllExceptListedIfNoMFA",
"Effect": "Deny",
"NotAction": [
"iam:CreateVirtualMFADevice",
"iam:EnableMFADevice",
"iam:GetUser",
"iam:ListMFADevices",
"iam:ListVirtualMFADevices",
"iam:ResyncMFADevice",
"sts:GetSessionToken",
"iam:ChangePassword",
"iam:GetAccountPasswordPolicy"
],
"Resource": "*",
"Condition": {
"Bool": {
"aws:MultiFactorAuthPresent": "false"
}
}
}A very good article already been published on AWS site which actually refers to reverse scenario where users should use MFA when using AWS CLI. Please refer the article here.
However, in most of the scenarios restricting CLI with MFA isn’t required, such as :
- When you are running some POC
- When another person applies the policy without estimating its full impact with half knowledge
- When you want to integrate with other DevOps/IAAC tools
Once you change the condition, just save the Policy. Please allow 2-3 minutes as sometimes the refresh is very slow. Usually it is done in seconds.
Now, lets try same command again. Try to decode the message
[root@automation-vm]# aws sts decode-authorization-message --encoded-message U9CRdJngzzYdO3DLoYRcjxlVqLcxtZ9D74GYS61GRdFbyasjEME4VDH9TqAwzzmTjkt_2v5HuGl90zw3XT7l_IEKedG2cR6MKUNy7N0-vj1YsdCk9wl5Lf0VyctWbzghsNWUNfBU5X2izecx3nJeGjJCygOCGTHklKrpxq8Ytex_WRHnOkiScj40KC0B46l9k0z2CtHRPzkcKC89Gp9JyDMuqpJ-LhyPzfA-hM1gIyloaXRkcH6t49_VNqssWht3vKW9zpLpnpNWILINWi7u4fgz8g4M3E1W1MHDLX365nA-Z3rmVl5f4zG6qLZ3Vsq9HzYhp5rjUJla-usjBzlxrUrlhlps2S4-OGSAdFUGClqAjAFAkdfBr4n0albOL1v7sZ0p6BYgPanySEl36tSY_uLpowzQeAhXk3dyktT-F-FBCn7GDj5dq39Wb0LsZP7pV5Qt0PfkMFaC9XN148RSfU18tXxyO6sShQoOu0Cv2BKy2HpyOpX0ZKTEEJEA6c1fehwhFWAFwoDOuPB4xgP8Jy0ETd63ERGxBv7qdAklpynrEEwp970dOnIvy7g5VB7M87oboBbyVqMc24hDrs6hAXDkS7lWxCw6oxq99OG4Vms2w1zUM5YUJYGWT82BsaFrP2QJGHZcHw5jfu1A0Ow4y0IoQviB5wB0A2c_vEAxkLche1oMkw87sy1YZDiaW4glj_6l7ZVSnW2Y6iF0PW8Qd3fezuCgV3LjY4TPJyR1OD1rL05TGocEHKJ
{
"DecodedMessage": "{\"allowed\":false,\"explicitDeny\":true,\"matchedStatements\":{\"items\":[{\"statementId\":\"DenyAllExceptListedIfNoMFA\",\"effect\":\"DENY\",\"principals\":{\"items\":[]},\"principalGroups\":{\"items\":[{\"value\":\"AG****************G7JX\"}]},\"actions\":{\"items\":[{\"value\":\"iam:CreateVirtualMFADevice\"},{\"value\":\"iam:EnableMFADevice\"},{\"value\":\"iam:GetUser\"},{\"value\":\"iam:ListMFADevices\"},{\"value\":\"iam:ListVirtualMFADevices\"},{\"value\":\"iam:ResyncMFADevice\"},{\"value\":\"sts:GetSessionToken\"},{\"value\":\"iam:ChangePassword\"},{\"value\":\"iam:GetAccountPasswordPolicy\"}]},\"resources\":{\"items\":[{\"value\":\"*\"}]},\"conditions\":{\"items\":[{\"key\":\"aws:MultiFactorAuthPresent\",\"values\":{\"items\":[{\"value\":\"false\"}]}}]}}]},\"failures\":{\"items\":[]},\"context\":{\"principal\":{\"id\":\"AI*****************X2\",\"name\":\"toolsadmin\",\"arn\":\"arn:aws:iam::57*********9:user/toolsadmin\"},\"action\":\"ec2:RunInstances\",\"resource\":\"arn:aws:ec2:eu-west-1:57********9:instance/*\",\"conditions\":{\"items\":[{\"key\":\"ec2:InstanceMarketType\",\"values\":{\"items\":[{\"value\":\"on-demand\"}]}},{\"key\":\"aws:Resource\",\"values\":{\"items\":[{\"value\":\"instance/*\"}]}},{\"key\":\"aws:Account\",\"values\":{\"items\":[{\"value\":\"57**********9\"}]}},{\"key\":\"ec2:AvailabilityZone\",\"values\":{\"items\":[{\"value\":\"eu-west-1c\"}]}},{\"key\":\"ec2:ebsOptimized\",\"values\":{\"items\":[{\"value\":\"false\"}]}},{\"key\":\"ec2:IsLaunchTemplateResource\",\"values\":{\"items\":[{\"value\":\"false\"}]}},{\"key\":\"ec2:InstanceType\",\"values\":{\"items\":[{\"value\":\"t2.micro\"}]}},{\"key\":\"ec2:RootDeviceType\",\"values\":{\"items\":[{\"value\":\"ebs\"}]}},{\"key\":\"aws:Region\",\"values\":{\"items\":[{\"value\":\"eu-west-1\"}]}},{\"key\":\"aws:Service\",\"values\":{\"items\":[{\"value\":\"ec2\"}]}},{\"key\":\"ec2:InstanceID\",\"values\":{\"items\":[{\"value\":\"*\"}]}},{\"key\":\"aws:Type\",\"values\":{\"items\":[{\"value\":\"instance\"}]}},{\"key\":\"ec2:Tenancy\",\"values\":{\"items\":[{\"value\":\"default\"}]}},{\"key\":\"ec2:Region\",\"values\":{\"items\":[{\"value\":\"eu-west-1\"}]}},{\"key\":\"aws:ARN\",\"values\":{\"items\":[{\"value\":\"arn:aws:ec2:eu-west-1:57**********9:instance/*\"}]}}]}}}"
}You can now try to run all the AWS CLI commands without any issue 🙂
All done – Happy Learning !!
thanks
it worked – thanks for sharing