We are using GuiXT to help simplify some screens for our clients users. When using pushbuttons to call other functions or transactions, we have found that we cannot return to the exact screen from which the new function/transaction was called. Now I would like to know if there is a way to allow the user to go back to the calling screen once he or she has finished with the transaction called up to use the GuiXT pushbutton.

It is currently not possible to invoke transactions in a true "call mode" from GuiXT. You can try the following:
(a) When you call the new transaction, use the /Oxxxx... instead of /Nxxxx. At the end of the your transaction the user can then close the new mode and continue with his original transaction
(b) Sometimes there exist menu items in the transaction that actually call the new transaction, returning to the screen again.
We will try to find a better solution. We already spoke with SAP development about implementing a /Cnnnn command, like /N or /O, that calls a transactions. But this will not be implemented by SAP in the near future, and we are currently testing a GuiXT call to an ABAP function that will then call the transaction. We hope that this will lead to a satisfying solution.