The retailers are investing in ways to court beauty consumers away from specialty retailers, trying new staffing models and remixing the merchandising assortment.