You can skip this part if you’re confident with programming with PyTorch.

https://colab.research.google.com/drive/12nQiv6aZHXNuCfAAuTjJenDWKQbIt2Mz#scrollTo=zzyM_Iz95bSb

<aside> 💡

A few things to watch out for:

• You can't do any in-place operations on a tensor that has requires_grad=True. (This prevents you from inadvertently mutating it in a way that isn't tracked for backprop purposes.)
• You also can't convert a tensor with requires_grad=True to numpy (for the same reason as above). Instead, you need to detach it first, e.g. y.detach().numpy().
• Even though y.detach() returns a new tensor, that tensor occupies the same memory as y. Unfortunately, PyTorch lets you make changes to y.detach() or y.detach.numpy() which will affect y as well! If you want to safely mutate the detached version, you should use y.detach().clone() instead, which will create a tensor in new memory.

RL Connection: You would want to be doing simulator-related tasks with numpy, convert to torch when doing model-related tasks, and convert back to feed output into simulator.

</aside>