In the inspector for the texture rect, you want to check "expand" for allowing the scale to be applied to the image inside the rect.
Don't use scale in the transform, just use the regular rectangle dimensions.
If you apply the scale to the transform it will affect all the children, while if you use the "expand" it will operate internally on the texture. Just adjust how you want to visualize the image inside the rect on the following drop-down menu.