ReInvent 2023: Trying out Pod Identity Agent

Out of re:Invent 2023 one of the features that caught my eye is the new EKS add on: Pod Identity Agent that promises to simplify IAM permissions for EKS, and a necessity to be sure, as IAM permission on EKS are amongst one of the main reasons my wall continues to gain depth as I bang my head against it whenever I have to deal with roles and permissions on AWS.

The present

Currently, the way it works if you don’t adopt the new feature — assuming no configuration is done on a brand-new provisioned cluster — is that you create your node group with a role. Whenever a pod authenticates to AWS, it does so with the role of the node. So, the role of the pod is equal to the role of the node. Let’s go through an example with a Deployment called api-service in the namespace api-service.

The problem is that if you host multiple pods on the same node, they all share the same IAM role, and your infrastructure doesn’t respect the principle of least privilege. An upgrade of this model was done at re:Invent 2022, which allows pods to assume a role via a service account

apiVersion: v1 
kind: ServiceAccount 
metadata: 
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/api-service 
  name: api-service 
  namespace: api-service

It’s a simple configuration, this maps it directly by ARN to a role you’ve created.

A key thing to note is the Trust Policy.

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Effect": "Allow", 
            "Principal": { 
                "Service": "ec2.amazonaws.com" 
            }, 
            "Action": "sts:AssumeRole" 
        }, 
        { 
            "Effect": "Allow", 
            "Principal": { 
                "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/${ID}" 
            }, 
            "Action": "sts:AssumeRoleWithWebIdentity", 
            "Condition": { 
                "StringEquals": { 
                    "oidc.eks.eu-west-1.amazonaws.com/id/${ID}":sub": "system:serviceaccount:api-service:api-service" 
                } 
            } 
        } 
    ] 
}

• The ${ID} comes from IAM > Identity Providers and you should see your cluster’s Identity Provider there.
• On the String Equals attribute the values are: "${OIDC_PROVIDER_URL}:sub" : system:serviceaccount:${NAMESPACE}:${NAME_OF_SERVICE_ACCOUNT}
• The principal is Federated Trust and you give it the OIDC provider URL.

This new model looks like this:

The problem with this approach is that it’s rather cumbersome to update.

If you want to add, for example, a user service on the user-service namespace, and make both of these service assume the same role, you’d have to change the trust policy to this.

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Effect": "Allow", 
            "Principal": { 
                "Service": "ec2.amazonaws.com" 
            }, 
            "Action": "sts:AssumeRole" 
        }, 
        { 
            "Effect": "Allow", 
            "Principal": { 
                "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/${ID}" 
            }, 
            "Action": "sts:AssumeRoleWithWebIdentity", 
            "Condition": { 
                "StringEquals": { 
                    "oidc.eks.eu-west-1.amazonaws.com/id/${ID}":sub": "system:serviceaccount:api-service:api-service", 
                    "oidc.eks.eu-west-1.amazonaws.com/id/${ID}":sub": "system:serviceaccount:user-service:user-service" 
                } 
            } 
        } 
    ] 
}

You get the idea, managing this with terraform, would mean your modules would need to accommodate a list and every time you update your service you’d also have to update the IAM role etc… Not Ideal.

The Future

The EKS Pod Identity simplifies this by introducing a new Domain for the Principal Identity, the pods.eks.amazonaws.com , updating the the role trust relationship, to this:

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Effect": "Allow", 
            "Principal": { 
                "Service": "pods.eks.amazonaws.com" 
            }, 
            "Action": [ 
                "sts:AssumeRole", 
                "sts:TagSession" 
            ] 
        } 
    ] 
}

Much cleaner! Notice how it’s not Federated Principal. We have upgraded from Federated to Domain access. That means we no longer deal with conditional operators and allowing services based on StringEquals conditions.

Secondly you have enable this as an addon on your EKS Cluster Console on the AWS Web Interface.

Once you do this you should see the pod created on your cluster via kubectl.

kubectl get pod --namespace kube-system -l app.kubernetes.io/name=eks-pod-identity-agent 
NAME                           READY   STATUS    RESTARTS   AGE 
eks-pod-identity-agent-tb8zs   1/1     Running   0          1m

And finally, to associate the pod with the role, you have to go to your EKS Cluster and go to the Access tab and in the Pod Identiy Associations . At this time, this has to be manually done, sadly no terraform modules currently support this new resource =( .

So for example let’s associate role: api-service-development-eu-west-1 to a service account

And that would be it! Your pod can now assume the role with no problems! . Assuming you gave it the correct permissions to do what you wanted but that’s a separate story!

Conclusion

It’s better than before; finally, I can stop hunting for the OIDC Provider URL every time I need a new role, as well as debugging string conditionals or missing ARNs in the trust policy, etc. However, it’s still not quite there yet. It would be great if the association was done through labels in a Kubernetes-native way. The next upgrade to this will be when the AWS provider in Terraform eventually supports this, allowing you to create automation around it. While it’s not perfect, abstracting away more and more IAM hassles in favor of streamlining the process of associating IAM roles with service accounts is a step in the right direction.