Unfortunately, I was not able to find a reference with "ready-to-use" formula to convert from RGB to RYB and back to RGB. The paper Paint Inspired Color Mixing and Compositing for Visualisation - Gossett and Chen describes the general idea of the RYB color model in the section "2 ACHIEVING INTUITIVE COLOR MIXING".