In the details of a team member setting, you can define his workload permissions and specify which other team members he will be able to see in the Workload view. 


Be aware that a team member will always be able to see himself.