Quadratic nonlinear frequency conversion demonstrates an intrinsic geometric phase, as its dynamics is equivalent to a two-level physical system. This unique property can be fully controlled by tailoring nonlinear photonic crystals, creating nonlinear geometric phase plates which also exhibit optical nonreciprocity. Here, we demonstrate both theoretically and experimentally, a robust adiabatic frequency conversion scheme that generates geometric-phase shaped beams in sum-frequency generation. We generate a Hermite-Gauss 01 mode and a cylindrical lens, with distinct nonreciprocity demonstrated in the latter. Our preliminary results pave the way towards all-optically controlled, non-reciprocal geometric phase plates in the frequency domain.